Adding 2nd data center to ES Service


We are currently only using ES to centralize our logs. We currently have 1 east coast data center and 1 deployment in the ES Service (cloud) for that data center. We are now adding a west coast data center and would like to centralize the logs from both data centers into the ES Service. The data centers currently do not have any high speed connection between them. Each data center will run Logstash to send data to the ES Service in the Cloud.

My questions are:

  1. Is it possible and/or recommended to configure both data centers to send data to the single deployment we currently have in ES Service?

  2. If we need to create a 2nd deployment, does that mean I need to alter my ES account in some way to allow for more that 1 deployment? Or am I allowed to create multiple deployments and I'm only charged according to how my space all the data from both data centers require?


Based on the high level picture you've given us, it seems fine for logs from both data centers to be sent to the same cloud ES service.

I assume (but you didn't actually say), that your current ES Service cluster is in a deployment region that is geographically close to your East DC. That means ingest from your current DC travels over relatively short, hopefully high speed network links and has high bandwidth and low latency. That's good but not required.

Ingest processes (in your case, pushing logs into ES) will typically deal quite well with higher latency as long as there is enough bandwidth to get the data through. I would be very surprised if your West DC is bandwidth-constrained when pushing data into our cloud.

What you don't want to do with ES (and our cloud ES service won't let you do this), is put cluster nodes in separate data centers. When ES nodes are talking to each other they are quite chatty, and cannot tolerate high-latency links. If you tried to run a cluster that had some nodes on the east coast, and some on the west coast, that would give you a very unsatisfactory experience. But sending data from 1 coast to the other is unlikely to be a problem.

What you haven't talked about is who queries this logging data and where they are located. I assume (because you didn't mention it) that there's no change there.
If you wer going to have east coast employees who always looked at east DC data and west coast employees who always looked at west DC data, then that might be an argument for separate cloud services for the 2 separate groups of employees.
But if all/most of your users want to look at all/most of the data, then having a single service will make more sense.

A single account may have multiple deployments. There are limits (so you don't accidentally run up really big bills), but if you head to you can chose to create a second (/third/fourth) deployment.

Our pricing model is more complex than simply "storage" (space), but if you create a second deployment then you just pay for the resources used by that deployment. Our pricing calculator can help work out those costs.

However, per my answer to your first question, I don't think you need a second deployment. It will mean your data is physically separate and you have to use separate Kibana instances and separate queries to find that data from each deployment.

Thanks Tim.

Your assumptions were all correct. Our ES Service cluster is in a deployment region that is geographically class to our East DC and for now only East employees will be looking at the West data. Good to know that I should be able to get both the East and West Logstash instances to send their data to the same deployment in our ES Service deployment.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.