Server configuration suggestions

Elastic Stack Elasticsearch
1

Hi all,

At work we are using elasticsearch to index the logs from network devices
like firewalls, proxy servers, dns servers etc. I was not involved in the
initial setup of the system however after issues with performance and
dropping logs I have made it my goal to get the configuration right. I
believe the root cause of our issues is the poor elasticsearch
configuration and I would like to ask the input from you guys to see how we
should configure everything.

Our server has 24 cores and 128 gb of ram. In terms of storage we
unfortunately have to make due with 10 network drives mapped on the system.
I imagine we can alter the number of mapped drives by simple readjusting
their sizes.

Our current configuration is a single ES instance with about 60gb of ram
assigned to it.

I want to set up a cluster of ES instances on the same machine. However,
this is where I would to get input from you guys. Considering the specs
mentioned above, what should we opt for? I am looking for recommendations
regarding the number of instances, the amount of memory to assign to each
of them, the disk layout and perhaps configuration settings.

Regarding the amount of instances and memory, I am considering running 4x
ES with 16GB each.

Regarding the storage, I am unsure whether it is even beneficial to divide
it into x pieces per ES considering we have to work with network storage.
If it is, we can probably divide each mapped drive's size in half, and
assign 5 mapped drives to each node.

Regarding settings, I would like to disable replication as I feel it does
not make a lot of sense in this setup. Furthermore our ES system is not a
true production server with high availability requirements and the logs are
kept on another system for compliance reasons anyway.

Any input would be highly appreciated!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a2f3e746-f299-43e2-bec7-1f826b434c17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Multiple instances per physical makes some sense here.

Start with 4 x 16GB heap instances (32GB system), use ES "striping" (ie
multiple path.data) to write to N drives per instance (probably 2, or 3 if
you can get more). Use the os.processors setting to lock 5 cores per
instance, leaving some for the OS to manage things with.

Disabling replication is your call here, most wouldn't recommend it, but
given you're also only running on a single physical you're not really
redundant anyway. However you can potentially utilise
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shadow-replicas.html

On 4 May 2015 at 00:30, Jerome Kleinen jkleinen@gmail.com wrote:

Hi all,

At work we are using elasticsearch to index the logs from network devices
like firewalls, proxy servers, dns servers etc. I was not involved in the
initial setup of the system however after issues with performance and
dropping logs I have made it my goal to get the configuration right. I
believe the root cause of our issues is the poor elasticsearch
configuration and I would like to ask the input from you guys to see how we
should configure everything.

Our server has 24 cores and 128 gb of ram. In terms of storage we
unfortunately have to make due with 10 network drives mapped on the
system. I imagine we can alter the number of mapped drives by simple
readjusting their sizes.

Our current configuration is a single ES instance with about 60gb of ram
assigned to it.

I want to set up a cluster of ES instances on the same machine. However,
this is where I would to get input from you guys. Considering the specs
mentioned above, what should we opt for? I am looking for recommendations
regarding the number of instances, the amount of memory to assign to each
of them, the disk layout and perhaps configuration settings.

Regarding the amount of instances and memory, I am considering running 4x
ES with 16GB each.

Regarding the storage, I am unsure whether it is even beneficial to divide
it into x pieces per ES considering we have to work with network storage.
If it is, we can probably divide each mapped drive's size in half, and
assign 5 mapped drives to each node.

Regarding settings, I would like to disable replication as I feel it does
not make a lot of sense in this setup. Furthermore our ES system is not a
true production server with high availability requirements and the logs are
kept on another system for compliance reasons anyway.

Any input would be highly appreciated!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a2f3e746-f299-43e2-bec7-1f826b434c17%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a2f3e746-f299-43e2-bec7-1f826b434c17%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9SgwSQKkKRjBS_Xgka_3UqOH-Z-__%2Bw-HGLx2V75WgfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.