ELK workflow with Filebeat - could you explain in detail?

Elastic Stack Beats
1

Hi buddy,
I'm very very confused and try to clear the flow of Elasticsearch - Logstash - Kibana .
Now , this is my situation : I wanna monitoring my Kafka server by ELK and i found the way ; use ELK . And this is my thinking flow

  • on kafka server : setup filebeat/metricbeat , beat will send log to logstash:5044
  • Logstash config input from beat > 5044 & output to Elasticsearch:9200
  • Kibana visualize the metric on Elasticsearch on Dashboard with Kibana:5601

but when i follow the offical Document of Filebeat :
/etc/filebeat/filebeat.yml

setup.kibana:
host: "localhost:5601"
output.logstash:
hosts: ["127.0.0.1:5044"]

I can not run : filebeat setup -e
and follow document said :

For this configuration, you must load the index template into Elasticsearch manually because the options for auto loading the template are only available for the Elasticsearch output.

after that , Kibana index have 2 file with format :
logstash-datetime-0001
filebeat-datetime-0001

1- Is this correct ? , how can i know which index is send by logstash , is it logstash.....0001, and i can delete filebeat....0001 ?

In Logstash config :
i just follow example :
input {
tcp {
port => 5044
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}

2 - How can i use filter - if i use filter , the index in Elasticsearch and Kibana still is filebeat.... or logstash.... or any new index ?, Do you have any example filter for Kafka?

thank you

2

quite simple : for the setup just ignore logstash and do a
metricbeat setup -E setup.kibana.host=your_kibana_host :5601 -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["your_elasticsearchhost:9200"]'
same with filebeat , both on the command line of the host you have the beat installed.
But make sure that you call the setup BEFORE you start the metricbeat or filebeat services for the first time, in other words: before you put any data to logstash->elasticsearch.

3

Hi bro,

just ignored logstash

you mean ignored the log when use command : filebeat setup -e?
and i have to use this

metricbeat setup -E setup.kibana.host=your_kibana_host :5601 -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["your_elasticsearchhost:9200"]'

okay, let's move. And in my elasticsearch have index name : logstash-...... , not filebeat..... right?

4

Even when send filebeat data through logstash, and even you apply some filter in logstash, you should only hava a "filebeat*" index in ES/Kibana

5

so what's about logstash-.....00001 index ?
i just put input & output in logstash conf file, not enable filter at this time

6

i guess you should delete the logstash index, it should not appear again. Just the filebeat index should be there.
see : Error in index or visualisations for "[Metricbeat System] Overview ECS"

use Kibana-> Devtools>Console:
and execute:
DELETE /filebeat-*
DELETE /_template/filebeat-7.3.1

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.