Elasticsearch Issue with custom json input data using logstash

Hello Everyone,

I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.

With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.

We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.

Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
exceptions, the performance really degrades and become unusable. The redis
will consume more and more memory. Elasticsearch will come to a point
where it is doing GC. Restarting Elasticsearch doesn't help.

The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.

It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.

I'm wondering if there are any recommendation on Elasticsearch and logstash
configuration.

Do I need to alter the logstash mapping?

Thanks,
Jared

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f9c03a63-41bc-4e73-99ec-3bf6b54cbbb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Also the CPU usage also jump considerably too.

Thanks,
Jared

On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com wrote:

Hello Everyone,

I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.

With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.

We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.

Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
exceptions, the performance really degrades and become unusable. The redis
will consume more and more memory. Elasticsearch will come to a point
where it is doing GC. Restarting Elasticsearch doesn't help.

The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.

It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.

I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.

Do I need to alter the logstash mapping?

Thanks,
Jared

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How are you feeding the json logs into ES?

On 19 February 2015 at 10:56, jlam@bills.com wrote:

Also the CPU usage also jump considerably too.

Thanks,
Jared

On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com
wrote:

Hello Everyone,

I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.

With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.

We are logging apache logs on all the web servers without any performance
issues. Kibana works fine and performance is pretty fast.

Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException:
failed to parse" exceptions, the performance really degrades and become
unusable. The redis will consume more and more memory. Elasticsearch will
come to a point where it is doing GC. Restarting Elasticsearch doesn't
help.

The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.

It seems that if Elasticsearch is having problem execute bulk item index,
it degrades the performance considerably.

I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.

Do I need to alter the logstash mapping?

Thanks,
Jared

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_nJqMnZSGFnw3h1LyEnxhX-17mwL7PoV1RHunhiymv5A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I'm using logstash.

On the client, I setup the input for logstash with json codec and output to
redis server. There is a logstash instance to pop from redis list into ES.

Thanks,
Jared

On Wednesday, February 18, 2015 at 5:14:15 PM UTC-8, Mark Walkom wrote:

How are you feeding the json logs into ES?

On 19 February 2015 at 10:56, <jl...@bills.com <javascript:>> wrote:

Also the CPU usage also jump considerably too.

Thanks,
Jared

On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, jl...@bills.com
wrote:

Hello Everyone,

I'm hoping I might get some help on how Elasticsearch. I'm getting
performance issues with Elasticsearch.

With our current setup:
We have Elasticsearch (1.4.3), redis, and logstash installed on the same
server with 30GB of memory. The ES_HEAP_SIZE is set to 15GB. Each server
has logstash installed and push the logs to redis. The logstash on the
server will pickup logs from redis and push them to Elasticsearch.

We are logging apache logs on all the web servers without any
performance issues. Kibana works fine and performance is pretty fast.

Here is the issue:
We want to do custom application logging. The logs are in json format.
When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException:
failed to parse" exceptions, the performance really degrades and become
unusable. The redis will consume more and more memory. Elasticsearch will
come to a point where it is doing GC. Restarting Elasticsearch doesn't
help.

The dataset is not that big comparing to others. The daily size of the
dataset is probably 2GB to 3GB of logs.

It seems that if Elasticsearch is having problem execute bulk item
index, it degrades the performance considerably.

I'm wondering if there are any recommendation on Elasticsearch and
logstash configuration.

Do I need to alter the logstash mapping?

Thanks,
Jared

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/608d2cb9-2e54-4d47-bf92-d50ca85ca2e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.