I have a problem with the data flow configuration:
Beat -> Logstash -> Elasticsearch
I initially configured everything in the flow:
Beat -> Elasticsearch
of course I initially did the required setup (adding standard visualizations/dashboards/templates, etc.).
When, for example, Metricbeat sent data directly to Elasticsearch everything worked OK.
In Kibana, for example, in Inventory/Metrics I can see the information regarding the nodes where Metricbeat is. Everything is ok.
However, when changing the flow with additionally Logstash, not everything works ok for me.
In Metricbeat, I changed the output from elasticsearch to logstash (to logstash per port).
In Logstash, however, I added a pipeline (input beat and output elasticsearch).
Below is the pipeline configuration in Logstash:
input {
beats {
port => 5045
}
}
output {
elasticsearch {
hosts => ["https://D-ELK1:9200", "https://D-ELK2:9200", "https://D-ELK3:9200"]
cacert => "/etc/logstash/certs/ca.crt"
user => "elastic"
password => "xxxx"
# index => ".ds-metricbeat-8.3.2-logstash-%{+YYYY.MM}"
index => "metricbeat-8.3.2-%{+YYYY.MM}"
# index => "%{[@metadata][beat]}-%{[@metadata][version]}"
# template_name => "metricbeat-8.3.2"
# template_overwrite => false
# data_stream => true
}
}
Some of the fields commented but I also tried with them (when they were uncommented).
Logstash and Metricbeat services are working (active).
I don't see any big errors in the logs.
I don't see the information displayed in Inventory/Metrics.
On the other hand, I can see in Discovery that data from this node (from Metricbeat) is available.
Why am I not seeing the data in Inventory/Metrics?
By default, Beats sends data to Elatsicsearch and this is processed in a data stream.
I don't know exactly what it looks like when I do it through the pipeline in Logstash.
How should I configure the elasticsearch output in Logstash pipeline for beats agents so that this is handled well?