Elasticsearch 1.7.5.1: delete document has no impact on the index size

[Antoine_Brun]

Hello,

we have a ES cluster 1.7 (it's production so we couldn' t upgrade it to a more recent version) and we are storing network device syslogs in daily indexes.
Because the volume of data collected is huge, we have scripts to manage the log retention policy: we keep security logs for 90 days but traffic logs (99% of the data) are deleted after 8 days.
To delete we use this kind of query:
2017-03-06 15:36:03: curl -XDELETE '127.0.0.1:9200/prodlogs-2017-02-27/logs/_query' -d '{
"query": {
"bool": {
"must": [{
"query_string": {
"query": "type:traffic",
"default_operator": "AND"
}
}, {
"range": {
"date": {
"lt": "now-8d"
}
}
}]
}
}
}'

so every day we delete 98-99% of the data in the index and the kopf plugin show that the number of document is reduced as expected.
The problem we have is that the size of the index is no decreased.

I have tried to optimize the index but the size stays the same.

This is a big problem for us because we are running out of disk space (4Tb).

Any idea here would be more that welcome

Antoine

[Christian_Dahlqvist]

I would recommend storing your two different types of data in two separate indices as they have different retention periods. This would allow you to simply delete complete daily indices rather than using delete by query. For the data that is in the current indices holding all types of data, it is probably easier and more efficient to reindex the security logs into separate indices as the volumes are low and then delete the large indices that contain traffic data.

[Antoine_Brun]

yes, that's a possibility we'll have to consider.
In the mean time, I just found out that I can use the option only_expunge_deletes when optimizing