Why this large differences

Elastic Stack Elasticsearch
1

** Situation **

Current

I issued this

GET /_cat/allocation?v&h=disk.indices,disk.avail,disk.used,disk.total,disk.percent

I get

disk.indices disk.avail disk.used disk.total disk.percent
        12gb      7.5gb    42.3gb     49.9gb           84

We are inserting 1000-2000 of documents per second for a total size of 16Gb per day ,(just to give you an idea),

We roll over indices at 500mb sizes

{ 
  "conditions": {
       "max_age": "2h", 
       "max_size": "524288000b"
   } 
}

We drop oldest indices if the 10Gb limit is reached

We have a problem that if we allow the index data to grow to 20G (we have 50Gdisk after all) it will hit the 85% threshold

Doesn elastic flush files that are unused?

2

I am not sure I follow what your problem is? What do you mean by flush files that are unused?