Beat -> Logstash -> Elasticsearch configuration problem

I have a problem with the data flow configuration:

Beat -> Logstash -> Elasticsearch

I initially configured everything in the flow:

Beat -> Elasticsearch

of course I initially did the required setup (adding standard visualizations/dashboards/templates, etc.).

When, for example, Metricbeat sent data directly to Elasticsearch everything worked OK.

In Kibana, for example, in Inventory/Metrics I can see the information regarding the nodes where Metricbeat is. Everything is ok.

However, when changing the flow with additionally Logstash, not everything works ok for me.

In Metricbeat, I changed the output from elasticsearch to logstash (to logstash per port).

In Logstash, however, I added a pipeline (input beat and output elasticsearch).

Below is the pipeline configuration in Logstash:

input {
  beats {
    port => 5045

output {
    elasticsearch {
        hosts => ["https://D-ELK1:9200", "https://D-ELK2:9200", "https://D-ELK3:9200"]
        cacert => "/etc/logstash/certs/ca.crt"
        user => "elastic"
        password => "xxxx"
        # index => ".ds-metricbeat-8.3.2-logstash-%{+YYYY.MM}"
        index => "metricbeat-8.3.2-%{+YYYY.MM}"
        # index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        # template_name => "metricbeat-8.3.2"
        # template_overwrite => false
        # data_stream => true

Some of the fields commented but I also tried with them (when they were uncommented).

Logstash and Metricbeat services are working (active).
I don't see any big errors in the logs.

I don't see the information displayed in Inventory/Metrics.

On the other hand, I can see in Discovery that data from this node (from Metricbeat) is available.

Why am I not seeing the data in Inventory/Metrics?

By default, Beats sends data to Elatsicsearch and this is processed in a data stream.

I don't know exactly what it looks like when I do it through the pipeline in Logstash.

How should I configure the elasticsearch output in Logstash pipeline for beats agents so that this is handled well?

Perhaps Look at the example on this page

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.