How to reduce the log size?

Elastic Stack Elasticsearch
1

Hi Team,

Sorry if this question is already answered. I have a serious problem, My each log in Elasticsearch is about 2 MB (My indices now are around 350 GB), I am confused Is that the expected behavior from Elasticsearch or some weird going on my application ?

Thanks in advance.

2

What is the total raw data volume you have ingested?

3

I am sorry, How can I check the data volume ?

4

What does the cat indices API say about this index/indices? How large are your raw documents?

5

/var/lib/elasticsearch/nodes/0/indices ----> 312G

I went into particular index then i have --> 0 1 2 3 4 _state (states)

/var/lib/elasticsearch/nodes/0/indices/03ZVSj4iQvW8sQziqfKfsg/0/index ---> i have docs here

where all the docs are around --> 260K

6

just to be clear i have 40 indices now which takes ~ 312G

7

What is the output of the API I asked about?

8

Below are the samples

Nov -2
green open logstash-2017.11.02 -thc3P_8TjmMUkkDntpKGQ 5 1 61648586 0 19.7gb 9.8gb

Nov -1
green open logstash-2017.11.01 tJJ3LosfTASQ3PBJ4x8_gw 5 1 65746826 0 21.9gb 10.9gb

9

It looks like each record takes up around 170 bytes per shard. If you are looking to reduce this I would recommend optimise your mappings as described in the documentation and this blog post.

10

Thanks for the info, will check the documention

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.