Looking for recommendations on elasticsearch set up for logging

Elastic Stack Elasticsearch
1

Hi,

I am looking for recommendations on spanning my cluster on cloud across multi region. Any sample ideas?

The problem statement is as below and i don't see much info on a set up like this:

-- Need to connect my apps hosted in 2 DC's to an ELK cluster in public cloud.
-- My applications are run completely on either our DC's or on cloud but we managed to run mostly on DC's.
-- The idea is to eventually move to cloud but at this time i should be able gather logs if i am running 50-50 traffic on DC and cloud.

I am looking for ideas on how i can host individual components of ELK stack to ship my logs.

2

Spanning a cluster across DCs/regions is not a good idea, as Elasticsearch can detect latencies as node disconnections. If you had zones within a single region, that is ok.

Otherwise, just use Beats to send to a single cluster in the cloud. If you use Elasticsearch Service then you get Kibana as well, which negates you having to worry about deploying it.

3

Hi Mark,

Thanks for taking time to respond, appreciate it!

Here is the Idea that we have come up with, please suggest if this would make up a good way of architecting the solution to the initial problem stated:

  1. 2 DC's on prem will have logstash instances in each, file beat ships logs to the logstash.
  2. 2 regions on cloud having one cluster in each region will have logstash instances in each cluster , file beat agent running in application container send logs to the logstash intances.
  3. Elastic search 1 cluster in each region have 3 master/data nodes - 1master in each zone (no separate data nodes). Note: Sizing has been decided according to our needs.
  4. 2 kibana instances for each region with a GLB infront for users/management/queries etc.
  5. One on prem logstash will send the logs to elastic cluster to the region closer to it and viceversa.
  6. shards and replication are default i.e 5primary Shards/1Replica of each primary shard.

In current set up we have a tribe node that manages the cluster. Is there a way to do the same for the above idea using CCR/CCS. Please suggest and also would like some recommendation on this arch.

Thanks!

4

That's a lot of redundancy. If that's what the business is asking for, then design to that.

TBH I would probably start with a single cluster in the cloud somewhere, then ship everything to that.

5

Attached is rough sketch to what it may look like for better understanding.!

To your note on the single cluster design that i have initially pitched to the biz and some how because of the nature of elastic search in the current set up(interms of data retention and availability ) the above thought have such redundancy came up.!

6

That all makes sense.

7

What replaces the concept of a tribe node?

CCS or CCR? I am a lil confused comparing the concepts and does it apply to my idea here?

8

CCS is cross cluster search.
CCR is cross cluster replication.

9

So neither is related to tribe node concept.. that was my question basically.

10

CCS replaces tribe.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.