I recently finished deploying an ES/Logstash cluster for a small
environment. It's a two-node local cluster but I'm getting horrible
performance and frequent crashes. I'll soon be standing up a cluster in
another environment that's roughly 10x the size of this first one so I've
got to figure out how to better optimize the clusters. Here's the current
resource and indexing stats:
8 CPU, 64GB RAM, 1TB storage
140 log sources, 21 indexed fields, ~35,000 message per minute, ~60GB/day
At present, index/search performance is awful. If I search for any time
period that's larger than a day or so, ES will usually crash and require a
manual restart. I'm going to be standing up a second dedicated ES system
and I'll be configuring one host for indexing and the second for searching.
I'll also be enabling the mmapfs store, disabling the _all field, and
disabling storing and/or indexing on some of the fields. At least that's my
plan so far -- it makes sense in my head but I'm not sure if it will
actually be the most efficient solution.
If I'm doing something stupid or if anybody has other recommendations, do
tell!
I recently finished deploying an ES/Logstash cluster for a small
environment. It's a two-node local cluster but I'm getting horrible
performance and frequent crashes. I'll soon be standing up a cluster in
another environment that's roughly 10x the size of this first one so I've
got to figure out how to better optimize the clusters. Here's the current
resource and indexing stats:
8 CPU, 64GB RAM, 1TB storage
140 log sources, 21 indexed fields, ~35,000 message per minute, ~60GB/day
At present, index/search performance is awful. If I search for any time
period that's larger than a day or so, ES will usually crash and require a
manual restart. I'm going to be standing up a second dedicated ES system
and I'll be configuring one host for indexing and the second for searching.
I'll also be enabling the mmapfs store, disabling the _all field, and
disabling storing and/or indexing on some of the fields. At least that's my
plan so far -- it makes sense in my head but I'm not sure if it will
actually be the most efficient solution.
If I'm doing something stupid or if anybody has other recommendations, do
tell!
On 11 February 2014 10:03, Harry Truman <land...@gmail.com <javascript:>>wrote:
I recently finished deploying an ES/Logstash cluster for a small
environment. It's a two-node local cluster but I'm getting horrible
performance and frequent crashes. I'll soon be standing up a cluster in
another environment that's roughly 10x the size of this first one so I've
got to figure out how to better optimize the clusters. Here's the current
resource and indexing stats:
8 CPU, 64GB RAM, 1TB storage
140 log sources, 21 indexed fields, ~35,000 message per minute, ~60GB/day
At present, index/search performance is awful. If I search for any time
period that's larger than a day or so, ES will usually crash and require a
manual restart. I'm going to be standing up a second dedicated ES system
and I'll be configuring one host for indexing and the second for searching.
I'll also be enabling the mmapfs store, disabling the _all field, and
disabling storing and/or indexing on some of the fields. At least that's my
plan so far -- it makes sense in my head but I'm not sure if it will
actually be the most efficient solution.
If I'm doing something stupid or if anybody has other recommendations, do
tell!
You write "ES will usually crash" - but how does it crash? Are there
messages in the log?
Do not use Java 7u51, it may cause trouble, 7u25 is known to be stable.
Why do you only use 12G heap if you have 64G RAM on a node? Why do you
limit your resources with ES_DIRECT_SIZE? Why do you use 5 shards per index
instead of 1 if you have 2 nodes?
With 0.90.11, Java 7u25, mmapfs, mlockall, 16G or 20G heap, 1 shard / 1
replica per index for 2 nodes, and unlimited ES_DIRECT_SIZE your system
will work better.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.