Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing and
retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage from
posgres to elastic search. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my system
when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service calls
to push logs data into ES. There are about 5000 records being inserted into
one index for every request into the application (all in JSON format).
Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications running
on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing and
retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage from
posgres to Elasticsearch. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my system
when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service calls
to push logs data into ES. There are about 5000 records being inserted into
one index for every request into the application (all in JSON format).
Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications
running on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Can that be optimized? Does that run real time? I configured
index.refresh_interval to run every 30 seconds. Can I optimize indexing
process anyway.
I have observed that whenever I process 3-4 events in my application
causing about 5-6K records insertion in elasticseach, there's cpu spike.
I monitored heap as well, and its way below (200mb) size configured (2gb)
in ES.
I don't need to use any fancy feature of cluster or availability at this
point.
My whole objective (from previous post) is to ensure that I store logs
information and retrieve it using API and show in my web administration
application without causing any performance or cpu spike.
Please advise.
Thanks!
Regards,
Sagar Shah
On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote:
That would be the indexing process that happens when you send the logs, as
ES needs to process and turn these into an inverted index for searching.
On 11 February 2015 at 07:23, Sagar Shah <sagars...@gmail.com
<javascript:>> wrote:
Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing and
retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage
from posgres to Elasticsearch. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my
system when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service
calls to push logs data into ES. There are about 5000 records being
inserted into one index for every request into the application (all in JSON
format). Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications
running on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Can that be optimized? Does that run real time? I configured
index.refresh_interval to run every 30 seconds. Can I optimize indexing
process anyway.
I have observed that whenever I process 3-4 events in my application
causing about 5-6K records insertion in elasticseach, there's cpu spike.
I monitored heap as well, and its way below (200mb) size configured (2gb)
in ES.
I don't need to use any fancy feature of cluster or availability at this
point.
My whole objective (from previous post) is to ensure that I store logs
information and retrieve it using API and show in my web administration
application without causing any performance or cpu spike.
Please advise.
Thanks!
Regards,
Sagar Shah
On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote:
That would be the indexing process that happens when you send the logs,
as ES needs to process and turn these into an inverted index for searching.
Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing and
retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage
from posgres to Elasticsearch. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my
system when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service
calls to push logs data into ES. There are about 5000 records being
inserted into one index for every request into the application (all in JSON
format). Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications
running on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Can that be optimized? Does that run real time? I configured
index.refresh_interval to run every 30 seconds. Can I optimize indexing
process anyway.
I have observed that whenever I process 3-4 events in my application
causing about 5-6K records insertion in elasticseach, there's cpu spike.
I monitored heap as well, and its way below (200mb) size configured (2gb)
in ES.
I don't need to use any fancy feature of cluster or availability at this
point.
My whole objective (from previous post) is to ensure that I store logs
information and retrieve it using API and show in my web administration
application without causing any performance or cpu spike.
Please advise.
Thanks!
Regards,
Sagar Shah
On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote:
That would be the indexing process that happens when you send the logs,
as ES needs to process and turn these into an inverted index for searching.
Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing and
retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage
from posgres to Elasticsearch. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my
system when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service
calls to push logs data into ES. There are about 5000 records being
inserted into one index for every request into the application (all in JSON
format). Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications
running on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Can that be optimized? Does that run real time? I configured
index.refresh_interval to run every 30 seconds. Can I optimize indexing
process anyway.
I have observed that whenever I process 3-4 events in my application
causing about 5-6K records insertion in elasticseach, there's cpu spike.
I monitored heap as well, and its way below (200mb) size configured
(2gb) in ES.
I don't need to use any fancy feature of cluster or availability at this
point.
My whole objective (from previous post) is to ensure that I store logs
information and retrieve it using API and show in my web administration
application without causing any performance or cpu spike.
Please advise.
Thanks!
Regards,
Sagar Shah
On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom
wrote:
That would be the indexing process that happens when you send the logs,
as ES needs to process and turn these into an inverted index for searching.
Hello everyone,
I am new to Elastic Search world and I am evaluating it for storing
and retrieval of my application logs, which is currently being stored in
postgres database.
For performance and disk space reasons, I plan to switch logs storage
from posgres to Elasticsearch. I plan to store log information for various
applications as different indices in ES.
But when I implemented this tool, I noticed lots of cpu spike on my
system when it received the request for inserting logs data into ES.
I am using Linux and allocated ES a 2GB RAM. I am using REST service
calls to push logs data into ES. There are about 5000 records being
inserted into one index for every request into the application (all in JSON
format). Refresh interval is kept to 30 seconds in elasticsearch.yml file
Number of shards configured to 1 and replicas to 0. There are about 20
index created for different applications, which might eventually grow upto
50 (as it is per application) Also, advise, if it should be separate index
per application, or single index storing logs of all of the applications. I
expect about 3k to 5k log messages per any request in any application
Single node is being used for storing logs of multiple applications
running on same node under separate JVM
JDK: 1.7
ES_HEAP_SIZE=2g
Also configured, mlockall to true in elasticsearch.yml file.
I am clueless in identifying root cause of cpu spike.
Can anyone please suggest a way out in further troubleshooting and
determining the root cause. What should be the next steps to troubleshoot
the same.
ES version: 1.4.2
Please let me know, if you need any further information.
Appreciate your inputs and help!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.