I'm using ES for searching for events based on date and geo distance, as
well as textual content. I'm also using logstash for handling app logging
and analytics.
I've noticed after I have millions of records from logging/analytics, the
events search starts slowing down.
I'm currently using just one node (testing all of this out before going
into production). The event data is currently stored in just one index.
My question is, what is a good way to handle this scenario to prevent event
searches from becoming slow? Should I use separate nodes just for
logging/analytics? Should I index the event data differently?
Also, I would be grateful if someone could point me to some good general
information about this kind of thing.
On Monday, October 6, 2014 4:28:31 AM UTC-4, Michael Irwin wrote:
I'm using ES for searching for events based on date and geo distance, as
well as textual content. I'm also using logstash for handling app logging
and analytics.
I've noticed after I have millions of records from logging/analytics, the
events search starts slowing down.
I'm currently using just one node (testing all of this out before going
into production). The event data is currently stored in just one index.
My question is, what is a good way to handle this scenario to prevent
event searches from becoming slow? Should I use separate nodes just for
logging/analytics? Should I index the event data differently?
On Mon, Oct 6, 2014 at 10:30 AM, Michael Irwin mdi@livej.am wrote:
Also, I would be grateful if someone could point me to some good general
information about this kind of thing.
On Monday, October 6, 2014 4:28:31 AM UTC-4, Michael Irwin wrote:
I'm using ES for searching for events based on date and geo distance, as
well as textual content. I'm also using logstash for handling app logging
and analytics.
I've noticed after I have millions of records from logging/analytics, the
events search starts slowing down.
I'm currently using just one node (testing all of this out before going
into production). The event data is currently stored in just one index.
My question is, what is a good way to handle this scenario to prevent
event searches from becoming slow? Should I use separate nodes just for
logging/analytics? Should I index the event data differently?
You could run less intense queries. Get more ram. Finally if io wait is a
problem then you could switch to/add more solid state disks. Or you can
add more nodes. We've done all of those for our Elasticsearch (no
Logstash/Kibana in front though).
On Mon, Oct 6, 2014 at 10:30 AM, Michael Irwin mdi@livej.am wrote:
Also, I would be grateful if someone could point me to some good general
information about this kind of thing.
On Monday, October 6, 2014 4:28:31 AM UTC-4, Michael Irwin wrote:
I'm using ES for searching for events based on date and geo distance, as
well as textual content. I'm also using logstash for handling app logging
and analytics.
I've noticed after I have millions of records from logging/analytics,
the events search starts slowing down.
I'm currently using just one node (testing all of this out before going
into production). The event data is currently stored in just one index.
My question is, what is a good way to handle this scenario to prevent
event searches from becoming slow? Should I use separate nodes just for
logging/analytics? Should I index the event data differently?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
