Logs is sending into Elasticsearch takes much more disk space than in txt format


(Emil Eldar Rustamov) #1

Dear all,

Logs is sending into Elasticsearch takes much more disk space than in txt format.

I have 20 application servers with ~ 5 apps on every host. So daily volume of logs ~ 10GB. But in ES it equals ~ 90-100GB.

How can i optimize it and reduce used disk space?

Quick response will be greatly appreciated.


(David Pilato) #2

You can first visit the mapping. It's by default generated automatically.
May be sometimes you don't need to index some fields or you don't want to compute aggregations?
May be you want to disable _all field which is enabled by default...


(Emil Eldar Rustamov) #3

Could you please give more detailed info about? Or provide some links where i can read about.

Thanks in advance, David


(David Pilato) #4

https://www.elastic.co/guide/en/elasticsearch/reference/5.4/mapping-all-field.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/mapping.html

Might help


(Christian Dahlqvist) #5

You can look at this blog post. It is for ES 2.x, but a lot of it is still valid. Another thing that can affect storage is how large shards you have. Compression tend to be more efficient for larger shards. What is your average shard size?


(Emil Eldar Rustamov) #6

I don't know.

How can i get know how large is my shard size? Is there any APIs?


(Christian Dahlqvist) #7

Use the _cat/indices or _cat/shards APIs.


(Emil Eldar Rustamov) #8

Here is my current statement of indices and shards:

curl -XGET 'http://localhost:9200/_cat/indices?v'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logs-2017.05.08 BDesFsuKTd-yX5MXb3CH-A 5 1 118281769 0 40.6gb 40.6gb
yellow open logs-2017.05.07 wEKExj52QKiUx71C0L9uJw 5 1 253644202 0 83.1gb 83.1gb
green open kubernetes-2017.04.14 rHGsoOtzRgOSOP5qOHDk7w 5 0 7101 0 972.4kb 972.4kb
green open .kibana _SKcDbJJQm6wXjzFac8_Ng 1 0 4 0 20.4kb 20.4kb
green open kubernetes-2017.04.13 67I6BJjbQNSp0f-dXnt8Ww 5 0 21618 0 2.5mb 2.5mb
green open kubernetes-2017.04.24 LwuM2ZFHQoCiGyxRa0mMsA 5 0 26016 0 3.6mb 3.6mb
green open kubernetes-2017.04.21 MLJWqs_3RE26Yoiok53SKQ 5 0 31683 0 5mb 5mb
green open kubernetes-2017.04.26 aZST0jKwSZ2oebZztq_hNg 5 0 28025 0 2.8mb 2.8mb

Here is my current statement of shards:

curl -XGET 'http://localhost:9200/_cat/shards?'

ubernetes-2017.04.14 2 p STARTED 1412 187.8kb 10.0.139.79 elastic
kubernetes-2017.04.14 4 p STARTED 1440 186.3kb 10.0.139.79 elastic
kubernetes-2017.04.14 3 p STARTED 1400 205.6kb 10.0.139.79 elastic
kubernetes-2017.04.14 1 p STARTED 1444 193.6kb 10.0.139.79 elastic
kubernetes-2017.04.14 0 p STARTED 1405 198.9kb 10.0.139.79 elastic
.kibana 0 p STARTED 4 20.4kb 10.0.139.79 elastic
logs-2017.05.08 2 p STARTED 23633110 8gb 10.0.139.79 elastic
logs-2017.05.08 2 r UNASSIGNED
logs-2017.05.08 1 p STARTED 23645002 8.2gb 10.0.139.79 elastic
logs-2017.05.08 1 r UNASSIGNED
logs-2017.05.08 3 p STARTED 23638619 8gb 10.0.139.79 elastic
logs-2017.05.08 3 r UNASSIGNED
logs-2017.05.08 4 p STARTED 23627459 8gb 10.0.139.79 elastic
logs-2017.05.08 4 r UNASSIGNED
logs-2017.05.08 0 p STARTED 23637948 8gb 10.0.139.79 elastic
logs-2017.05.08 0 r UNASSIGNED
kubernetes-2017.04.26 2 p STARTED 5578 581.7kb 10.0.139.79 elastic
kubernetes-2017.04.26 1 p STARTED 5610 571.4kb 10.0.139.79 elastic
kubernetes-2017.04.26 3 p STARTED 5674 586.3kb 10.0.139.79 elastic
kubernetes-2017.04.26 4 p STARTED 5536 545.5kb 10.0.139.79 elastic
kubernetes-2017.04.26 0 p STARTED 5627 601kb 10.0.139.79 elastic
kubernetes-2017.04.24 2 p STARTED 5265 789.6kb 10.0.139.79 elastic
kubernetes-2017.04.24 4 p STARTED 5310 749.7kb 10.0.139.79 elastic
kubernetes-2017.04.24 1 p STARTED 5092 759.2kb 10.0.139.79 elastic
kubernetes-2017.04.24 3 p STARTED 5184 715.9kb 10.0.139.79 elastic
kubernetes-2017.04.24 0 p STARTED 5165 736.3kb 10.0.139.79 elastic
kubernetes-2017.04.13 2 p STARTED 4296 539.4kb 10.0.139.79 elastic
kubernetes-2017.04.13 1 p STARTED 4340 513.7kb 10.0.139.79 elastic
kubernetes-2017.04.13 4 p STARTED 4378 537.6kb 10.0.139.79 elastic
kubernetes-2017.04.13 3 p STARTED 4279 502.1kb 10.0.139.79 elastic
kubernetes-2017.04.13 0 p STARTED 4325 547kb 10.0.139.79 elastic
logs-2017.05.07 2 p STARTED 50724224 16.6gb 10.0.139.79 elastic
logs-2017.05.07 2 r UNASSIGNED
logs-2017.05.07 1 p STARTED 50732285 16.6gb 10.0.139.79 elastic
logs-2017.05.07 1 r UNASSIGNED
logs-2017.05.07 3 p STARTED 50719560 16.6gb 10.0.139.79 elastic
logs-2017.05.07 3 r UNASSIGNED
logs-2017.05.07 4 p STARTED 50732660 16.6gb 10.0.139.79 elastic
logs-2017.05.07 4 r UNASSIGNED
logs-2017.05.07 0 p STARTED 50735473 16.6gb 10.0.139.79 elastic
logs-2017.05.07 0 r UNASSIGNED
kubernetes-2017.04.21 2 p STARTED 6270 1mb 10.0.139.79 elastic
kubernetes-2017.04.21 1 p STARTED 6295 1mb 10.0.139.79 elastic
kubernetes-2017.04.21 3 p STARTED 6323 1mb 10.0.139.79 elastic
kubernetes-2017.04.21 4 p STARTED 6325 1004.3kb 10.0.139.79 elastic
kubernetes-2017.04.21 0 p STARTED 6470 1mb 10.0.139.79 elastic

Is there way to optimize shards or indices?

And how i can delete replica from indices?


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.