Sending data to Multiple elasticsearch containers

Elastic Stack Elasticsearch
1

Hi all,
I am having 3 docker containers of elasticsearch running on a single host on different ports, Now I want to send data to all the 3 containers so that I don't have any single point of failure . What are the different strategies of sending data ?

2

Welcome!

Normally clients are supporting multiple nodes so they will be able to switch to the next node and isolate the faulty node if needed.

What is sending the data to elasticsearch ?

3

That is what I want to know, what should I use to send data to elasticsearch containers .

4

What are you doing with Elasticsearch? Logs? Search feature from your application? Which programming language?

5

I have to store the logs on multiple containers so that if one goes down, I have the backup.

6

What tool is sending logs to elasticsearch?

7

Could you please clarify

Are these "containers"

  1. different nodes that belong to 1 elasticsearch cluster?

  2. each is it one 1 single node elasticsearch cluster?

8

Yes, these are 3 containers of elasticsearch .

  1. Yes, these 3 nodes belong to one cluster.
  2. Nope, it is a 3 node elasticsearch cluster
10

So since the nodes are all part of the same cluster.

  1. assuming you use the default of 1 primary and 1 replica shards the data will automatically be distributed across the nodes and the primary and replica shards will never be on the same node so you will have HA with respect if the data if you lose 1 container. Elasticsearch does this automatically for you.

  2. what I described above will happen even if you send the data to one node because Elasticsearch us a distributed data store. Do not try to send copies of same data to each node.

  3. for HA of sending the data to the cluster depending on the tool / which you have not told us the are methods to "load balance" the sending of the data.

Which tool are you using to send the data.

11

I'm using Logstash and logstash is getting data via filebeat.

12

So have a look at Elasticsearch output plugin | Logstash Reference [7.12] | Elastic

It allows to pass multiple hosts.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.