Help to setup Metricbeat to push system/application metrics to ES cluster

[jsoriano (Jaime Soriano)]

Hi @Arunlal_A,

@Arunlal_A

Please suggest the best way / practices to achieve this. Following are the options I found:

1. Set Metricbeat output to ES (Not preferring, as it's in different subnet)
2. Set Metricbeat output to Logstash.
3. Set Metricbeat output to Redis.

If possible I'd suggest setting ES as output. If the only problem with this option is the network separation there is a solution. You can add coordinator nodes to the networks where metricbeat is running and make these nodes the only ones that have access to the network where your Elastic Stack is running. This way you don't need to add additional elements, and it is also easier to setup metricbeat when it sends events directly to Elasticsearch.

Logstash, and Redis (or Kafka) can also be valid options, but I'd only use them if you have some other requirements or more complex architectures.

@Arunlal_A

• Any way to set the Metricbeat itself to collect data from Metricbeat in agent servers? Like some input mechanism?

I am not sure of understanding this question.

• If you mean to monitor metricbeat itself, take a look to Metricbeat monitoring.
• If you mean to make Metricbeat listen for custom events, take a look to the HTTP Server metricset.
• If you mean to monitor your own software, you can also take a look to the APM solution.

@Arunlal_A

I also tested the Metricbeat "internal queue" mechanism (queue.spool) with ES as output. Will this work for all other output type (like Logstash, Redis etc)?

Yes, this queue is independent of the output, events are stored in this queue before being sent to the outputs.