Help with Setting up new Index/ dashboards Filebeat 7.1.1 and logstash indexes

Elastic Stack Beats
1

Dear Support forum,

I have upgraded my ELK to 7.1.1 and wanted to use the plugins and dashboards that filbeat ships with natively however I am unable to get the index to stay consistant.

When using file beat and the setup command, it created a default index of filebeat-7.1.1-$DATE
but the issue with thhis is it has no roll up jobs / ILM functionality.

So I kick off logstash with my ES output as :

output {
elasticsearch { hosts => ["10.70.104.50:9200"]
manage_template => true
template_overwrite => true

and it automatically creates a uindex of logstash-$DATE-00000
and also sets a rollup job

So I figure If I set the template in my logstash output to "filebeat" instead of logstash it would create the rollup jobs but it fails to link the alias and thus doesnt work.

I have tried to override it with my command :

filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['10.70.104.50:9200'] -E setup.kibana.host=10.70.104.55:80 -E output.elasticsearch.index="logstash-%{+yyyy.MM.dd}" -E setup.template.pattern="logstash-*" -E setup.template.name="logstash-"

but it still gives me the filebeat index instead :frowning:

right now filebeat sends data to logstash and i get all my logs in kibana but the dashes dont work
since it wants a index pattern of filebeat-* vs the one I use logstash-*

2

@Lee_Lilleorg-Meilleu
I am not sure if I understand your problem correctly. Is it right that you want to index data from logstash into elasticsearch on the same index as from the filebeat index?

If yes, then you can set in logstash elasticsearch output plugin the attribute indexwith the value of the filebeat index alias on elasticsearch. And I think that filebeat got the default alias: filebeat-{agent.version}

So try this in logstash output

output {
    elasticsearch { 
      hosts : ["10.70.104.50:9200"]
      index : "filebeat-7.1.1"
    }
}

So that the logstash output is into the same index as the other filebeats are indexing data.
Doc to index attribute in logstash output plugin elasticsearch

3

Yup, the idea is to use the same index as what the filebeat setup command creates . though for what ever reason. using the filbeat created indexes , it wont do index managemnt / create a new one each day
switching the index on the logstash output -> Elasticsearch to use a specific index (instead of the default) makes it incompatible with the policies

4

So you have filebeats indexing data directly to elasticsearch. And you have filebeats indexing data through logstash and then to elasticsearch.
If you want to have the data which is piped through logstash to land in the same index as filebeat indices. you would need to specify he index. Maybe there is an another way. But I don't know it yet.

Yes if the filebeats are running with ILM then it will not create each day a new index. It will create an if the policy allows this. And the basic policy is something around 50GB and 7 days. so it will create a new index if the index gets 50GB big or it is 7 days old. Doc

Otherway would be to disable ILM, but it is a pretty nice feature which can manage your indices very well.

Not sure if i helped you =)

5

the idea is that it would all be one main index :

filebeat -> logstash -> es

but the initialization of the dashboards has to happen on filebeat->es
so I pick a client running file beat and execute the setup.

: index.lifecycle.name in template to {filebeat-7.1.1 map[policy:{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}]} as ILM is enabled.

after the setup: is creates the filebeat-7.1.1-$YYYY-mm-dd-00001 INDEX
and the alias filebeat-7.1.1

then I need to point the ES in the logstash pipeline to use the same index as the alias name?
instead of trying to make it filebeat-7.1.1-YYYY-MM-dd ?

6

Yes, you need to point the ES in logstash pipeline to the same alias name which has been defined by filebeat setup ilm. That is the way how we use it and this seems to work fine.
Let me know if this suits you as well.

7

Perfect, the Alias was what I was missing.

so filebeat creates the index / alias and ES just needs to be pointed to the alias ... Thanks for clarifying that. !!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.