Setup Elastic search cluster for production

Elastic Stack Elasticsearch
1

Hi

I am trying to setup elastic search cluster for production. I have approximately 500 vms which we will be monitoring. I am expecting 2gig data every day. I will be installing Metric beats and files beats on servers.

I want to save 2 years worth of data and backup rest to a mount and delete on the servers.

I am planning of using 2 vms with 4 cpu and 400gb of diskspace. One will be master and the other will be node.

Is there a configuration in Elastic search where I can mention that only 2 years worth of data to remain on the vm and backup remaining to a mount and delete on the server.

2

You are gonna log 2GB per day for 2 years. To 2 nodes with only 800GB of disk between them .. That does not compute. Also never use an even number of nodes. Use at least 3 nodes in a cluster to avoid split brain. So please rework your design to avoid unpleasentries when you go into production.

When it comes to your question about deleting old data: Yes, you can do that. Would suggest that you look into Curator.

3

So how does it work. The data is shared between three nodes in a cluster. And if I have one master and two nodes. what happens if master goes down. Does the data automatically get in to the other nodes which are online. unfortunately I can only use two vms with 800gb max. I will try to limit the data to one year.

does curator automatically delete the data?

4

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

Also note that you need a 3rd node for production. It can be a lighter machine with a master only node though.

You need to test how much space an index takes. Note that you should have no more than 50gb data per shard and a maximum of 20 shards per gb of HEAP. That should give you how much data you can keep online.

That said frozen indices feature could help to store more with the price of being slower when searching in frozen indices.

HTH

5

@staodd Thanks for your reply.

6

@dadoonet Thanks for your reply.

If I only run elastic search on a vm can I use 2 small vms with "1 cpu /400gb" data or will it impact the performance of elastic search. Currently I am only monitoring user traffic from filebeat and system performance via metricbeat. OR do i need large vms?

Can I have two different settings to delete logs. for filebeat data I want to retains for 2 years.
But where as data from metricbeats might not be useful so want to retain only data for 6 months.

7

It's no problem to have different retention on metric- and filebeat, keeping them in separate indices makes it easier. As we already said, you need at least 3 nodes for a shared nothing cluster. Performance depends on what you are going to use it for. Only storage, or are you gonna analyze as well? Remember you want at least 1 replica of your data for redundancy, which means you will use 2x the storage.
Id suggest you go back to the drawing board for a bit. Make a proper design, including redundancy/performance requirements. Also think about peak messages per second and size of a message. Those are just a few of the things that will have an impact on your choice of HW. Start with a top level design and work your way down. And DON'T do as you did now: skip to the details and ask for advice without giving us much to work with.

8

I'd also recommend looking at the Rollup API. If you just need "stats" on your old indices and don't need a fine granularity you will be able to compress a lot your data. That will help you to keep your retention period using a smaller amount of resources.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.