Filebeat (with ILM) -> Logstash -> Elasticsearch not working

Elastic Stack Beats
,
1

Hi,

I think I am missing some simple settings here.
I have enabled the ILM in beats:

setup.ilm.enabled: true
setup.ilm.rollover_alias: "log-dev-filebeat"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat_dev_policy"

output.logstash:

  hosts: ["XXXXXX:2222"]
  index: log-dev-filebeat

The logstash file has a Elasticsearch output section:

output
{
	elasticsearch
	{
			hosts => "XCCCCC:34534"
			index => "%{[@metadata][beat]}-%{[@metadata][version]}"
			timeout => 600
			user => 'TTTTTT'
			password => 'PPPPPP'
	}
}

Now the setup is working in the sense that index is created but I see that the index created is log-dev-filebeat-7.7.0. It is not followed by the 000001.

Any ideas?

2

If you have Logstash in the middle, then your output needs to be pointing to the rollover alias.

3

I am a bit confused on the output. Is it output section of logstash file? Or in the filebeat.yml?
I tried with changing the index in the logstash elasticsearch output section to the roll-over alias. But did not work. Do I have to do some bootstrapping thing which many people have been talking about?

output
{
	elasticsearch
	{
			hosts => "XCCCCC:34534"
			index => "log-dev-filebeat"
			timeout => 600
			user => 'TTTTTT'
			password => 'PPPPPP'
	}
}
4

Yes that one.

What do you mean by that?

5

I got an index log-dev-filebeat.

Also GET _cat/aliases did not list any alias getting created.

6

Have u run filebeat setup yet? You need to run the setup using th elasticsearch output and then u can push to logstash with the rollover alias. See Use ingest pipelines for parsing | Logstash Reference [7.13] | Elastic

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.