Hi,
I have a cluster, from which I want to send logs to Elasticsearch and show it in Kibana.
My question is - can I install on every server (in the cluster) the Filebeat agent and send the log directly to Elasticsearch (it's on another server)? Or do I have to send the logs into a Logstash and then forward the data from the Logstash into the Elasticsearch?
If it is possible to send it directly, how should I configure the Filebeat agents? Right now I have just set the path for the log file and the Elasticsearch server address. But the data in Kibana look messed up a bit... And it looks like not all the agents send the data successfully...
Another question - if I setup the Elasticsearch cluster in the future, should I somehow configure the Filebeat agents for it? Or will the IP address still be only one and the Elasticsearch cluster will solve it internally?
Thank you very much for the answeres, I have read the documentation and google this "problem" as well, but I'm still a little bit lost.
Milo.
You can send directly to Elasticsearch, logstash isn't required (unless you need/want the extra processing that LS provides). The relevant docs are here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration.html
But the data in Kibana look messed up a bit...
Could you elaborate? How? You'll have to configure your Kibana dashboard to use the correct fields that Filebeat emits, etc
Another question - if I setup the Elasticsearch cluster in the future, should I somehow configure the Filebeat agents for it? Or will the IP address still be only one and the Elasticsearch cluster will solve it internally?
If you add more nodes to an existing Elasticsearch cluster, you don't need to do anything. As long as the configured node(s) in the Filebeat config are still valid, the cluster will internally balance data across all the nodes.
However, if you add a new cluster, you'll need to reconfigure Filebeat to point at that new cluster.
Hi,
thank you very much for the reply. I've managed to make it work - only Filebeat + Elasticsearch. Now the data seems ok 
Thanks again, Milo.