Elastic Search - How to delete old records and improve performance

Elastic Stack Elasticsearch
1

Hi all,

I have a production box running old ES "0.20.3" .. I need help in cleaning
data back from 2012 ... because although I moved the whole indexs to a
machine with 6 SSD drives ... queries got better .. but still I need to
make this thing faster

is there a *safe *way to clean (choose what to clean based on date) and
improve the performance?

cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOG9y1VndKdaZTt19PN2aVobL5_-02QctZ_8t44VXuGG9azrqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

2

anything?...

On Monday, March 30, 2015 at 11:56:46 PM UTC+3, AALISHE wrote:

Hi all,

I have a production box running old ES "0.20.3" .. I need help in
cleaning data back from 2012 ... because although I moved the whole indexs
to a machine with 6 SSD drives ... queries got better .. but still I need
to make this thing faster

is there a *safe *way to clean (choose what to clean based on date) and
improve the performance?

cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e9d34bef-f899-40e5-82d6-8b74b3adf8d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

This is a community based list and support is provided by volunteers as
their time permits. If you want timely responses then I'd recommend you
sign up for some support, we'd be happy to have you as a customer
Subscriptions | Elastic Stack Products & Support | Elastic :slight_smile:

Is all the data in the one index, or are they time based?
If you need ES to be faster then I'd suggest you upgrade, 0.20 is no longer
supported and there are a lot of performance improvements in later
versions.

On 31 March 2015 at 11:17, AALISHE aalishe@gmail.com wrote:

anything?...

On Monday, March 30, 2015 at 11:56:46 PM UTC+3, AALISHE wrote:

Hi all,

I have a production box running old ES "0.20.3" .. I need help in
cleaning data back from 2012 ... because although I moved the whole indexs
to a machine with 6 SSD drives ... queries got better .. but still I need
to make this thing faster

is there a *safe *way to clean (choose what to clean based on date) and
improve the performance?

cheers!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e9d34bef-f899-40e5-82d6-8b74b3adf8d4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e9d34bef-f899-40e5-82d6-8b74b3adf8d4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Y3SF%2BwO%2BQMrUpc3Aj46Fx85xRyRpBMqcQ-Xxe0tLQdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi Mark,

thanks for taking the time to reply

1- yes I believe all the data is in one index .. I had attached the
index_status output ... and a screenshot for the head_plugin output
2- I am considering the upgrade to latest version .. but would that impose
a risk (if any)?

thanks again
cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d5529d52-0a21-402a-8265-78da41b2b05f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Can you create a new index and exclude what you dont need. we did this
recently because of some other mapping reason. take a snapshot (with newwer
version) before doing so.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c473d836-50dd-4706-b7ab-da29cacd2612%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

mkBig ... thanks for the suggestion ... but how do I exclude the things I
dont need ?

cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/454830bc-a9c5-4803-ae95-8b37f65fedbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

mkBig ... thanks for the suggestion ... but how do I exclude the things I
dont need ?

cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/64a09808-e8f6-4588-959b-8855a851ed03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

You could do a few things, from my perspective. (Hope this hits some idea's
you like)

  • upgrade, never hurts :slight_smile: as long as you read the release notes to make
    sure nothing your depending on.
  • Add some filters to your existing queries to exclude unneeded data

  • you can create "filter aliases" which lets you set up predefined alias
    with filtered terms in-case you can't delete or loose data

    Index Aliases | Elasticsearch Guide [1.3] | Elastic
    • I would rotate your index so all new data would go in to a new one
      while the archive can wither and die
    • Maybe rotate it on a daily basis or a point of time that works for
      your data sets you can still have basic aliases to summarize multiple days
      depending on your query needs
    • Use a project called curator to purge old data on a routine basis
      (cron)
  • GitHub - elasticsearch-dump/elasticsearch-dump: Import and export tools for elasticsearch & opensearch it does full dumps
    and also Query Style dumpes.
    • Dump the whole index
    • Then dump with a query
    • Delete and import your query dump to a Date based index that can be
      purged later by curator
  • Oh just thought of this one, increase the number of shards and ES
    nodes, I see you have only 2
    • To do this will require you to export and import the data again.
      (at least for the old data

On Tuesday, March 31, 2015 at 4:13:07 PM UTC-4, AALISHE wrote:

mkBig ... thanks for the suggestion ... but how do I exclude the things I
dont need ?

cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4228dbd3-84a1-41ba-bed9-3bcf4cb70033%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.