Disk space filled up with ES indices

Elastic Stack Elasticsearch
1

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before my
monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/logstash-2013.04.06'"

Are there ways to compress these indices, so that i can store more of them.
?
Is there a way to setup rotation ?

Thanks
Sunny

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Try to keep more fields as non-analyzed if possible. For that, you'll have
to define custom mapping.

Have a look at

Regarding rotation, search on google for a python script to delete older
indices.

On Wed, Apr 10, 2013 at 4:53 PM, Sunny Jaisinghani <
jaisinghani.sunny@gmail.com> wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before
my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/logstash-2013.04.06'"

Are there ways to compress these indices, so that i can store more of
them. ?
Is there a way to setup rotation ?

Thanks
Sunny

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Abhijeet.

Could you please give me some examples.
will it make any difference if i set

index.query.default_field to @message

in elasticsearch.yml

On Wednesday, 10 April 2013 16:53:48 UTC+5:30, Sunny Jaisinghani wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before
my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/logstash-2013.04.06'"

Are there ways to compress these indices, so that i can store more of
them. ?
Is there a way to setup rotation ?

Thanks
Sunny

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Setting default field won't help you in any way.

You can take the template I use as an example.
https://gist.github.com/shadyabhi/5354133 Although, you might not want to
use it as is because it's made for the kind of logs I use, for ex, the use
of uax_url_email tokenizer.

If you don't disable analyzing for fields you don't need to be analyzed,
you are just increasing the size of tokens and thereby increase the size of
index.

On Wed, Apr 10, 2013 at 5:45 PM, Sunny Jaisinghani <
jaisinghani.sunny@gmail.com> wrote:

Thanks Abhijeet.

Could you please give me some examples.
will it make any difference if i set

index.query.default_field to @message

in elasticsearch.yml

On Wednesday, 10 April 2013 16:53:48 UTC+5:30, Sunny Jaisinghani wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before
my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/**elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/**logstash-2013.04.06http://localhost:9200/logstash-2013.04.06
'"

Are there ways to compress these indices, so that i can store more of
them. ?
Is there a way to setup rotation ?

Thanks
Sunny

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks Abhijeet.

Could you please give me some examples.
will it make any difference if i set

index.query.default_field to @message

in elasticsearch.yml

Also, I was wondering if i can set the settings from the below link in yml
format in elasticsearch.yml

On Wednesday, 10 April 2013 17:15:31 UTC+5:30, Abhijeet Rastogi wrote:

Try to keep more fields as non-analyzed if possible. For that, you'll have
to define custom mapping.

Have a look at
GitHub - elastic/logstash: Logstash - transport and process your logs, events, or other data

Regarding rotation, search on google for a python script to delete older
indices.

On Wed, Apr 10, 2013 at 4:53 PM, Sunny Jaisinghani <jaisingh...@gmail.com<javascript:>

wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before
my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/logstash-2013.04.06'"

Are there ways to compress these indices, so that i can store more of
them. ?
Is there a way to setup rotation ?

Thanks
Sunny

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

IMO, you are probably barking up the wrong tree here - how much disk space
do you have available? ... don't put /opt/ in / , use /mnt ...or a properly
sized EBS...

we hit the practical limits of ES on our sized instances after about 3
months of data ( ~ 250K msgs / hour )...keeping it trimmed to ~ 1 month and
it is super fast.

On Wed, Apr 10, 2013 at 10:30 PM, Sunny Jaisinghani <
jaisinghani.sunny@gmail.com> wrote:

Thanks Abhijeet.

Could you please give me some examples.
will it make any difference if i set

index.query.default_field to @message

in elasticsearch.yml

Also, I was wondering if i can set the settings from the below link in yml
format in elasticsearch.yml

Logstash Elasticsearch Template · GitHub

On Wednesday, 10 April 2013 17:15:31 UTC+5:30, Abhijeet Rastogi wrote:

Try to keep more fields as non-analyzed if possible. For that, you'll
have to define custom mapping.

Have a look at https://github.com/logstash/**logstash/wiki/Elasticsearch-
**Storage-Optimizationhttps://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization

Regarding rotation, search on google for a python script to delete older
indices.

On Wed, Apr 10, 2013 at 4:53 PM, Sunny Jaisinghani <jaisingh...@gmail.com

wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash and
kibana.

After i setup up all 3 of them, everything was good for few days, before
my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/**elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/**logstash-2013.04.06http://localhost:9200/logstash-2013.04.06
'"

Are there ways to compress these indices, so that i can store more of
them. ?
Is there a way to setup rotation ?

Thanks

Sunny

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi Sunny,
Please keep the mailing list in the CC so this discussion is shared with
everyone .

I think the _template config becomes part of your cluster metadata and
you should not lose it. Having said that, I never had to do a full cluster
restart (that I recall)... easy to test/ confirm though :wink:

On Fri, Apr 12, 2013 at 5:24 PM, sunny jaisinghani.sunny@gmail.com wrote:

Thanks again Noberto
Do I have to do this everytime elasticsearch is restarted ?
If i load the settings as you suggest, where are these stored. ??

On Fri, Apr 12, 2013 at 12:37 PM, Norberto Meijome numard@gmail.comwrote:

Hi Sunny,
i don't think you can do that in the elasticsearch.yml - they are index
specific (well, template which will apply to an index), and they are
applied at real time.

Put the json in a file, then load it with

curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d
@logstash-template.json

On Fri, Apr 12, 2013 at 4:46 PM, Sunny Jaisinghani <
jaisinghani.sunny@gmail.com> wrote:

Thanks Noberto
Yes. AN EBS would solve my disk issues.

However, I want to include the following settings into my
elasticsearch.yml.
I am encountering JSON for the first time.
I am not sure how to get these settings into elasticsearch.yml
These will help indexing on "message" field, enable compression and will
let the other fields be "not analyzed"

{
"template": "logstash-*",
"settings" : {

    "number_of_shards" : 1,
    "number_of_replicas" : 0,
    "index" : {

        "query" : { "default_field" : "@message" },
        "store" : { "compress" : { "stored" : true, "tv": true } }

    }
},
"mappings": {
    "_default_": {

        "_all": { "enabled": false },
        "_source": { "compress": true },

        "dynamic_templates": [
            {
                "string_template" : {

                    "match" : "*",
                    "mapping": { "type": "string", "index": "not_analyzed" },

                    "match_mapping_type" : "string"
                 }
             }
         ],

         "properties" : {
            "@fields": { "type": "object", "dynamic": true, "path": "full" },

            "@message" : { "type" : "string", "index" : "analyzed" },

            "@source" : { "type" : "string", "index" : "not_analyzed" },

            "@source_host" : { "type" : "string", "index" : "not_analyzed" },

            "@source_path" : { "type" : "string", "index" : "not_analyzed" },

            "@tags": { "type": "string", "index" : "not_analyzed" },

            "@timestamp" : { "type" : "date", "index" : "not_analyzed" },

            "@type" : { "type" : "string", "index" : "not_analyzed" }

        }
    }
}

}

On Wednesday, 10 April 2013 19:41:31 UTC+5:30, Norberto Meijome wrote:

IMO, you are probably barking up the wrong tree here - how much disk
space do you have available? ... don't put /opt/ in / , use /mnt ...or a
properly sized EBS...

we hit the practical limits of ES on our sized instances after about 3
months of data ( ~ 250K msgs / hour )...keeping it trimmed to ~ 1 month and
it is super fast.

On Wed, Apr 10, 2013 at 10:30 PM, Sunny Jaisinghani <
jaisingh...@gmail.com> wrote:

Thanks Abhijeet.

Could you please give me some examples.
will it make any difference if i set

index.query.default_field to @message

in elasticsearch.yml

Also, I was wondering if i can set the settings from the below link in
yml format in elasticsearch.yml

https://gist.github.com/**deverton/2970285https://gist.github.com/deverton/2970285

On Wednesday, 10 April 2013 17:15:31 UTC+5:30, Abhijeet Rastogi wrote:

Try to keep more fields as non-analyzed if possible. For that, you'll
have to define custom mapping.

Have a look at https://github.com/logstash/**lo**
gstash/wiki/Elasticsearch-Storage-Optimizationhttps://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization

Regarding rotation, search on google for a python script to delete
older indices.

On Wed, Apr 10, 2013 at 4:53 PM, Sunny Jaisinghani <
jaisingh...@gmail.com> wrote:

Hello,

I recently setup ES 0.20.6 on a AWS EC2 instance along with logstash
and kibana.

After i setup up all 3 of them, everything was good for few days,
before my monitoring started complaining about the disk usage.

Currently, my filesystem can only store indices of 4 days.

ll /opt/elasticsearch/data/elasticsearch/nodes/0/indices/

total 16
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 6 17:00
logstash-2013.04.07
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 7 17:00
logstash-2013.04.08
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 8 17:00
logstash-2013.04.09
drwxr-x--- 8 elasticsearch elasticsearch 4096 Apr 9 17:00
logstash-2013.04.10

I am deleting the indices manually, since couple of days using "curl
-XDELETE 'http://localhost:9200/**logstas**h-2013.04.06http://localhost:9200/logstash-2013.04.06
'"

Are there ways to compress these indices, so that i can store more
of them. ?
Is there a way to setup rotation ?

Thanks

Sunny

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/**grou**ps/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Norberto 'Beto' Meijome

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.