I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.
With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.
We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.
Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
exceptions, the performance really degrades and become unusable. The redis
will consume more and more memory. Elasticsearch will come to a point
where it is doing GC. Restarting Elasticsearch doesn't help.
The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.
It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.
I'm wondering if there are any recommendation on Elasticsearch and logstash
configuration.
On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com wrote:
Hello Everyone,
I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.
With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.
We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.
Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
exceptions, the performance really degrades and become unusable. The redis
will consume more and more memory. Elasticsearch will come to a point
where it is doing GC. Restarting Elasticsearch doesn't help.
The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.
It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.
I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.
On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com
wrote:
Hello Everyone,
I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.
With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.
We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.
Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException:
failed to parse" exceptions, the performance really degrades and become
unusable. The redis will consume more and more memory. Elasticsearch will
come to a point where it is doing GC. Restarting Elasticsearch doesn't
help.
The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.
It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.
I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.
On the client, I setup the input for logstash with json codec and output to
redis server. There is a logstash instance to pop from redis list into ES.
Thanks,
Jared
On Wednesday, February 18, 2015 at 5:14:15 PM UTC-8, Mark Walkom wrote:
How are you feeding the json logs into ES?
On 19 February 2015 at 10:56, <jl...@bills.com <javascript:>> wrote:
Also the CPU usage also jump considerably too.
Thanks,
Jared
On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com
wrote:
Hello Everyone,
I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.
With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.
We are logging apache logs on all the web servers without any
performance issues. Kibana works fine and performance is pretty fast.
Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException:
failed to parse" exceptions, the performance really degrades and become
unusable. The redis will consume more and more memory. Elasticsearch will
come to a point where it is doing GC. Restarting Elasticsearch doesn't
help.
The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.
It seems that if Elasticsearch is having problem execute bulk item
index, it degrades the performance considerably.
I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.