Can Logstash manage Beats ILM and Templates?

Elastic Stack Logstash
1

For various reasons I can't have my Beats talking directly to Elasticsearch on all the networks I will be monitoring. So I need to use Logstash as a proxy. Beats will ship to Logstash, Logstash will ship to Elasticsearch.

I have this working. The part I don't have working is the Beats "setup" for ILM and Templates.

The docs for the Elasticsearch output plugin make it clear that Logstash can manage ILM and Templates. The part I'm stuck on is how do I tell Logstash to use the same settings the Beats would?

That said, it occurs to me this might be solvable just by placing the needed Beats on my Elasticsearch nodes, and giving those specific Beats the ability to output directly to ES. That assumes I don't somehow need all instances of my Beats to run the setup tasks individually. Would that work?

When searching, I found this unanswered topic: ILM Support for beat-logstash-elastic that is very similar to my question. I thought I'd mention it here.

2

So, after loading the beats dashboards into Kibana, I was getting all kinds of odd errors. Stuff about fields not being right for searching, or something like that.

I'm 90% sure that was due to me telling Logstash to manage templates for the beats data that passes through it.

I poked around at trying to fix it, but didn't get anywhere.

So I just wiped out all my data, and started from scratch.

This time I made sure to use Metricbeat, Filebeat, and Journalbeat on my Elasticsearch nodes to manage the templates, ILM settings, and dashboards. I had to make sure those specific instances wouldn't send via logstash, and then configured the Elasticsearch output for them.

I then stopped all data from being sent to ES, and then ran each beat's setup command.

Then I turned the ES node beats back on, and it looked like things were working properly.

After that I made sure Logstash isn't managing templates or ILM, then turned it back on so the rest of my hosts beats could continue shipping data.

I think all that should get my settings managed properly.

Anyone see anything I might have missed?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.