Beat -> Logstash -> Elasticsearch configuration problem

Elastic Stack Beats
1

I have a problem with the data flow configuration:

Beat -> Logstash -> Elasticsearch

I initially configured everything in the flow:

Beat -> Elasticsearch

of course I initially did the required setup (adding standard visualizations/dashboards/templates, etc.).

When, for example, Metricbeat sent data directly to Elasticsearch everything worked OK.

In Kibana, for example, in Inventory/Metrics I can see the information regarding the nodes where Metricbeat is. Everything is ok.

However, when changing the flow with additionally Logstash, not everything works ok for me.

In Metricbeat, I changed the output from elasticsearch to logstash (to logstash per port).

In Logstash, however, I added a pipeline (input beat and output elasticsearch).

Below is the pipeline configuration in Logstash:

input {
  beats {
    port => 5045
  }
}

output {
    elasticsearch {
        hosts => ["https://D-ELK1:9200", "https://D-ELK2:9200", "https://D-ELK3:9200"]
        cacert => "/etc/logstash/certs/ca.crt"
        user => "elastic"
        password => "xxxx"
        # index => ".ds-metricbeat-8.3.2-logstash-%{+YYYY.MM}"
        index => "metricbeat-8.3.2-%{+YYYY.MM}"
        # index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        # template_name => "metricbeat-8.3.2"
        # template_overwrite => false
        # data_stream => true
  }
}

Some of the fields commented but I also tried with them (when they were uncommented).

Logstash and Metricbeat services are working (active).
I don't see any big errors in the logs.

I don't see the information displayed in Inventory/Metrics.

On the other hand, I can see in Discovery that data from this node (from Metricbeat) is available.

Why am I not seeing the data in Inventory/Metrics?

obraz
obraz1662×362 32.6 KB

By default, Beats sends data to Elatsicsearch and this is processed in a data stream.

I don't know exactly what it looks like when I do it through the pipeline in Logstash.

How should I configure the elasticsearch output in Logstash pipeline for beats agents so that this is handled well?

2

Perhaps Look at the example on this page