Metricbeat - logstash - redis - elasticsearch

Elastic Stack Beats
1

Hello,

I am trying to use Metricbeat for a few days ago.

I did installed Metricbeat on a windows server with "memory" metricsets only.

Metricbeat is sendind logs to a Logstash server and Logstash is sending logs to Redis.
I can see Metricbeat logs in Redis.

Then I have another Logstash server which is taking Redis logs to load them in Elasticsearch.
I don't see anything in Elasticsearch.

I did add metricbeat.template.json but still nothing.

Thanks for your help.

Best regards
Antoine

2

Here is the output configuration

output {
        if "metricbeat" in [tags] {
                elasticsearch {
                        hosts => [ "172.17.0.22:9200" ]
                        index => "idx-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
                        flush_size => 100
                        idle_flush_time => 1
                        template => "/etc/logstash/templates.d/metricbeat.template.json"
                        template_name => "metricbeat"
                        template_overwrite => true
                        document_type => "logs"
                }
        }
        else {
                elasticsearch {
                        hosts => [ "172.17.0.22:9200" ]
                        index => "idx-%{meta_app}-%{meta_env}-%{+YYYY.MM.dd}"
                        flush_size => 100
                        idle_flush_time => 1
                        template => "/etc/logstash/templates.d/default-template.json"
                        template_name => "default"
                        template_overwrite => true
                        document_type => "logs"
                }
        }
}
3

BTW Metricbeat can send data directly to Redis with its redis output.

Do you know if Logstash is reading from redis? Can you add a non-conditional console output so you can see the events.

output {
  stdout { codec => rubydebug }
}

Do those events have a "metricbeat" tag?

Are there any errors in your LS logs or in your ES logs?

Because you have modified the destination index from the default, did you modify the index template to match the new index name?

4

Hello,

Thanks for your answer.

I just tried to debug logstash (the second one) with output stdout but no event appeared so Logstash is not reading from redis.

I already had a "metricbeat" tags in metricbeat configuration
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with eachstrong text
# transaction published.
#tags: ["tampon"]
tags: ["metricbeat", "METAS:env=staging,app=rfid,mod=central,com=jetty,log=access,fmt=GBM_RFID_ACCESS_JETTY,ml=n,hot=y,path=/var/log/central/access.log"]

5

it is better since I choose to directly send metricbeat to redis instead of going through logstash first.

Now I can see logs in logstash, I am going to try to push those logs to Elasticsearch.

Thanks a lot for your help Andrew

6

And now I can see Metricbeat logs in Elasticsearch, that's perfect !

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.