Determine size of logstash indexes in elasticsearch


(Tim Dunphy) #1

Hi guys,

I need to try to determine how much size logstash is taking up in indexing the logs.

We have a 3 node elastic search cluster with logstash indexes on each one corresponding to the same days.

i.e.:

on ES node 1:

953M logstash-2015.06.16
235M logstash-2015.06.17

on ES node 2:

240M logstash-2015.06.16
916M logstash-2015.06.17

on ES node 3:

1.2G logstash-2015.06.16
1.1G logstash-2015.06.17

So my question is, is the right approach to total all of the sizes from the 3 nodes for each day? So I would add the size of logstash-2015.06.17 on ES node1 to the sizes of the same index day for ES node 2 and ES node 3?

Or is there another way I need to do that?

Thanks,
Tim


(Aaron Mildenstein) #2

The cat API will do what you're looking for:

$ curl localhost:9200/_cat/indices?v
health status index               pri rep docs.count docs.deleted store.size pri.store.size
green  open   kibana-int            1   1          9            0    111.4kb         55.7kb
green  open   .marvel-2015.06.16    1   1     303466            0    655.7mb        327.1mb
green  open   .marvel-2015.06.13    1   1     303461            0    637.3mb        318.2mb
green  open   logstash-2015.06.09   5   1   15542546            0      2.6gb          1.3gb
green  open   .marvel-2015.06.09    1   1     304025            0    701.4mb        350.7mb
green  open   .marvel-2015.06.05    1   1     304371            0    691.3mb        345.6mb
green  open   logstash-2015.06.08   5   1   15531583            0      2.6gb          1.3gb
       close  logstash-2015.06.03
       close  .marvel-2015.06.03
green  open   logstash-2015.06.13   5   1   15534682            0      5.5gb          2.7gb
green  open   logstash-2015.06.04   5   1   15697405            0      2.6gb          1.3gb
green  open   .marvel-2015.06.14    1   1     303596            0    644.6mb        321.6mb
green  open   logstash-2015.06.11   5   1   15534964            0      2.6gb          1.3gb
green  open   logstash-2015.06.10   5   1   15543394            0      2.6gb          1.3gb
green  open   .marvel-2015.06.07    1   1     304367            0    694.1mb          347mb
green  open   .marvel-2015.06.10    1   1     304334            0    706.6mb        353.3mb
green  open   logstash-2015.06.15   5   1   15536330            0      5.5gb          2.7gb
green  open   .marvel-2015.06.17    1   1     201947            0    457.6mb        223.5mb
green  open   .marvel-2015.06.11    1   1     304978            0    703.5mb        351.6mb
green  open   logstash-2015.06.12   5   1   15535711            0      5.5gb          2.7gb
green  open   logstash-2015.06.16   5   1   15552236            0      5.8gb          2.9gb
green  open   logstash-2015.06.05   5   1   15771832            0      2.7gb          1.3gb
green  open   logstash-2015.06.06   5   1   15525431            0      2.6gb          1.3gb
green  open   logstash-2015.06.07   5   1   15530311            0      2.6gb          1.3gb
green  open   .marvel-kibana        1   1          1            0      6.5kb          3.2kb
green  open   .marvel-2015.06.04    1   1     304508            0      685mb        342.5mb
green  open   .marvel-2015.06.08    1   1     304442            0    697.9mb        348.9mb
green  open   .marvel-2015.06.15    1   1     303569            0    650.1mb          324mb
green  open   .kibana               1   1         22            1    101.5kb         50.7kb
green  open   .marvel-2015.06.06    1   1     304399            0    693.3mb        346.6mb
green  open   logstash-2015.06.14   5   1   15527486            0      5.5gb          2.7gb
green  open   .marvel-2015.06.12    1   1     303534            0    629.6mb        314.2mb
green  open   logstash-2015.06.17   5   1   10235701            0      3.4gb          1.6gb

(Aaron Mildenstein) #3

Hmmm. Without the "verbose," which is including the header, it fits here better:

$ curl localhost:9200/_cat/indices
green open  kibana-int          1 1        9 0 111.4kb  55.7kb
green open  .marvel-2015.06.16  1 1   303466 0 655.7mb 327.1mb
green open  .marvel-2015.06.13  1 1   303461 0 637.3mb 318.2mb
…

(Tim Dunphy) #4

Cool Aaron!! Seems really easy this way. Thanks!! That's what I'll do. :slight_smile:

Tim


(system) #5