How to delete docs.deleted from ELK?

[Yuri_Pires]

Hello, in this logstash index I used the delete_by_query endpoint to clean old logs from storage, I was successful in this step, but I found that the docs are still on the HD and I want to delete them to free up space.

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash                        adZNy34GTRu-sLGfl6jMDQ   1   1  476661826      7623516     58.4gb         58.4gb

I tried running the following command but it was unsuccessful to free up space:

curl -XPOST "http://localhost:9200/logstash/_forcemerge?only_expunge_deletes=true" -u elastic:xxxx

After that, I tried to run the command:

 curl -XPOST "http://localhost:9200/logstash/_forcemerge?max_num_segments=1" -u elastic:xxx

And the error was:

{"_shards":{"total":2,"successful":0,"failed":1,"failures":[{"shard":0,"index":"logstash","status":"INTERNAL_SERVER_ERROR","reason":{"type":"illegal_state_exception","reason":"this writer hit an unrecoverable error; cannot complete forceMerge","caused_by":{"type":"i_o_exception","reason":"No space left on device","suppressed":[{"type":"i_o_exception","reason":"No space left on device","suppressed":[{"type":"i_o_exception","reason":"No space left on device"},{"type":"i_o_exception","reason":"No space left on device","suppressed":[{"type":"i_o_exception","reason":"No space left on device"},{"type":"i_o_exception","reason":"No space left on device","suppressed":[{"type":"i_o_exception","reason":"No space left on device"}]}]}]}]}}}]}}

Has anyone gone through something similar to help me?

[PodarcisMuralis (Murat K.)]

It seems that you are facing a "No space left on device" error when trying to force merge the index in Elasticsearch. This error occurs because the storage device where Elasticsearch is storing its data has run out of space.

When you perform a force merge on an index with the max_num_segments=1, Elasticsearch tries to reduce the number of segments by merging them together. However, this process requires additional disk space for temporary storage during the merging process, and in your case, there is not enough free space on the device to complete the operation.

So i would try df on linux to check how much space I have. Then I would free up some space after that, try force merge again.

curl -XPOST "http://localhost:9200/logstash/_forcemerge?max_num_segments=1" -u elastic:your_password