_snapshot data takes up disk space

Elastic Stack Elasticsearch
1

Hello. We're observing strange behavior.

Elasticsearch appears to be storing some snapshot data directly in it's data directory. And there's quiet a lot of space occupied by this data

~> find /var/lib/elasticsearch/nodes/0/indices/ -name "_snapshot" |head -n 10
/var/lib/elasticsearch/nodes/0/indices/AdKyiLFoQHO_8GBTV8TrOA/15/_snapshot
/var/lib/elasticsearch/nodes/0/indices/A50S21ZJR3meXQOH9_QZaQ/4/_snapshot
/var/lib/elasticsearch/nodes/0/indices/sm4kdbzOSGe9TFDT1RkP2Q/2/_snapshot
/var/lib/elasticsearch/nodes/0/indices/7UsDXgVdQ7aQYuUNPSVGFg/10/_snapshot
/var/lib/elasticsearch/nodes/0/indices/Zq-xl4o1SeKmzHdm_ctPYA/1/_snapshot
/var/lib/elasticsearch/nodes/0/indices/evzgeSdYQMeVPYlDs7Zh-w/9/_snapshot
/var/lib/elasticsearch/nodes/0/indices/evzgeSdYQMeVPYlDs7Zh-w/29/_snapshot
/var/lib/elasticsearch/nodes/0/indices/HaKVrbd1QEudW9YPXOTPGw/1/_snapshot
/var/lib/elasticsearch/nodes/0/indices/Q2OB_y0MSmyskXqN1x_jvQ/7/_snapshot
/var/lib/elasticsearch/nodes/0/indices/ufWMN6HvSZStyNu4hfyx2w/0/_snapshot

~> find /var/lib/elasticsearch/nodes/0/indices/ -name "_snapshot" -type d -exec du {} \; | awk '{ SUM += $1} END { print SUM }'               
253066032  #241 GB

This data are not included in data_store and we're running out of space because of it.
I'm failing to find any information about it.

We're running self-hosted Elasticsearch 6.8, with free X-Pack license and no plugins.

2

This appears to be a reason https://github.com/elastic/elasticsearch/issues/50231

3

@makarchuk

You're correct. Currently the best workaround I can suggest is to not use source only, but rather use normal snapshots and remove those directories manually.