So I've collected about 100GB of logstash logs over 3 months.
So there are roughly 100 Indexes such as logstash-2014.07.01 and so forth.
I have a cluster of 3 EC2 Instances, 1 CPU w/ 4GB RAM Each. Granted it's
When I do queries, it's usually fast, until I run a large timespan that
span say 10-30 indexes. At that point, I'm guessing each node has loaded
so much index and field data that it was nearly impossible to avoid
overrunning the heap or RAM on the cluster. I end up with nodes at 100%
CPU, and 75% RAM usage.
I just wanted to check what was possible with tuning:
Given limited RAM, is it possible somehow tune my nodes such that in
event of a large query requiring too much RAM:
1a) The job gets killed due to timeout
1b) Something else saves my node from becoming non-responsive?
Is it possible to make some indexes work fast, while others slow?
2a) When I query historical data, I don't need an answer quickly. Just
2b) When I query the last 72 hours, I really want an answer quickly, even
if that means killing other jobs
Is it an unavoidable fact that as my data increases, I have no choice
but to either:
3a) Increase cluster RAM to hold every index/field at the same time?
3b) Delete indexes until everything fits in RAM?
As I attempt to open opensearch to more people, they are running queries in
Kibana that span a larger and larger timeframe. Thus leading to random
If there was just some way to prevent frozen nodes (Maxed out CPU @ 100%
despite ram usage at say 3gb out of 4gb) then I would have a more stable
As EC2 does carry a noticable cost, I was trying to minimize my EC2
requirement. So I'm trying to find ways to selectively reduce performance
where I don't need it.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0883a19c-48e5-46d0-9c69-3dc9f411c2e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.