Balance Between Heavy Indexing and Searching

Elastic Stack Elasticsearch
1

Hi,

I have a cluster of 20 nodes, 1 TB/day of data indexed, right now we only
keep the last 3 days opened but the customer wants us to open 6 months of
indexes.

We don't care about query execution times but only that the indexing
throughput wouldn't get hurt.
Is there anything we can do in the current installation without expanding
it with many more nodes?

What is the best way to approach this problem?

Thanks you so much,
David

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b281e94a-1bce-4bb9-adf3-3174054ec043%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

The best way is to add more nodes.
There isn't much you can do with that amount of data!

On 7 January 2015 at 06:09, David Mavashev cryptic86@gmail.com wrote:

Hi,

I have a cluster of 20 nodes, 1 TB/day of data indexed, right now we only
keep the last 3 days opened but the customer wants us to open 6 months of
indexes.

We don't care about query execution times but only that the indexing
throughput wouldn't get hurt.
Is there anything we can do in the current installation without expanding
it with many more nodes?

What is the best way to approach this problem?

Thanks you so much,
David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b281e94a-1bce-4bb9-adf3-3174054ec043%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b281e94a-1bce-4bb9-adf3-3174054ec043%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_HtCkWMLRh_jSfOe6f1HG-SoVuJVrE-h%2BJaXq_qm5KZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

That is a ton of data to keep open. Can you squish it somehow?

On Tue, Jan 6, 2015 at 3:24 PM, Mark Walkom markwalkom@gmail.com wrote:

The best way is to add more nodes.
There isn't much you can do with that amount of data!

On 7 January 2015 at 06:09, David Mavashev cryptic86@gmail.com wrote:

Hi,

I have a cluster of 20 nodes, 1 TB/day of data indexed, right now we only
keep the last 3 days opened but the customer wants us to open 6 months of
indexes.

We don't care about query execution times but only that the indexing
throughput wouldn't get hurt.
Is there anything we can do in the current installation without expanding
it with many more nodes?

What is the best way to approach this problem?

Thanks you so much,
David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b281e94a-1bce-4bb9-adf3-3174054ec043%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b281e94a-1bce-4bb9-adf3-3174054ec043%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_HtCkWMLRh_jSfOe6f1HG-SoVuJVrE-h%2BJaXq_qm5KZA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_HtCkWMLRh_jSfOe6f1HG-SoVuJVrE-h%2BJaXq_qm5KZA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd07EW9Or9E0f8R6Dh0M%3DMGuWgRPUchcePSUcYvthctFbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.