Shipping logs from Elasticsearch nodes to Elasticsearch cluster

Elastic Stack Elasticsearch
1

Is there a recommended way to ship the logs produced by Elasticsearch nodes over to the Elasticsearch cluster itself for centralized logging purposes?

Thanks in advance for any input

2

Elasticsearch Nodes = Master + Data + Worker + Coordinating Nodes
Elasticsearch Cluster = Combination of the above

So, I didn't understand the question.. Could you be more specific ?

3

Hi Jack, sure.
My team has an Elasticsearch Cluster used for centralizing the logs produced by different services and applications we own.
We would like to include also the logs produced by the different nodes that are part of this same Elasticsearch cluster, more specifically, the Elasticsearch server logs. The main goal is to increase observability and simplify debugging issues that sometimes happen in the cluster.

4

Add two(for HA) more Logstash nodes for log collection in the cluster itself. And Two (for HA)Logstash pointing to the elasticsearch. So now you will have 4 logstash (two in one subnet and the other two in a different subnet) nodes pointing to your elasticseach.

5

Thanks for your input Jack. We are currently not using Logstash and our setup relies entirely on Filebeat and ingest nodes with certain pipelines for the most exotic log formats, as most of the logs are in JSON anyway. Is there a way we can ingest Elasticsearch logs without resorting to use Logstash? Perhaps by configuring the Elasticsearch logs to use JSON formatting?

6

Hi @sterago,

You can send elasticsearch node logs to elasticsearch cluster using filebeat.

here is the filebeat confihuration for elastic output:

#-------------------------- Elasticsearch output ------------------------------

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]

      # Optional protocol and basic auth credentials.
      protocol: "https"
      username: "changeme"
      password: "changeme"

      ssl.enabled: true

Hope this helps.

Thanks,
Shrikant

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.