Compress and copy elasticsearc indexes


(Roberto) #1

Hi Guys!

My elasticsearch indexes are filling the disk. I would like to compress these indexes. Is it possible to compress these indexes and copy to another computer in a way that I can restore when needed?

Would anyone help me?


(Luca Wintergerst) #2

Hi Roberto,
take a look at Snapshot and Restore
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

This is our way of doing backups.
While these have exactly the same storage size as your original data, you can compress them afterwards using any compression tool of your choice.
This will take a long time and will slow down your restore (as it's not implemented into elasticsearch and has to be done separately) - so my advice would be to just start with snapshots without compressing them afterwards


(Mark Walkom) #3

We compress data in ES by default, you can increase the default compression level though - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/tune-for-disk-usage.html#_use_literal_best_compression_literal


(Roberto) #4

Hi @lwintergerst, i am new with elasticsearch. I know very little about elasticsearch, but by reading the page you have already published, I understand that it is only possible to take a snapshot of all indexes.

What I wanted to know is if, it is possible to back up only one or a few days of the indexes, for example, let's say I have the whole March month in logs. But I just want to back up the "logstash-2017.04.01", "logstash-2017.04.02" and "logstash-2017.04.03" indexes. I want to archive them on another machine and delete them from the machine where the elasticsearch is (this I know how to do, rsrs, "curl -XDELETE 'localhost: 9200 / logstash-2017.04.02'") but I want to keep the indexes archived as a backup form in another machine, so that I can someday restore it if necessary.

Sorry for the English, I do not speak very well. I hope you understood.

Thanks for the help!


(Roberto) #5

Hi @warkolm, thanks for the reply, but I do not know where I set this up. Is it in elasticserach.yml? It's just that I do not know much about elasticsearch.


(Luca Wintergerst) #6

Hello Roberto,
This is what you are looking for. It is also part of the documentation that I linked above :slight_smile:
This will only back up "index_1" and "index_2"

PUT /_snapshot/my_backup/snapshot_1
{
  "indices": "index_1,index_2",
}

(Roberto) #7

:confounded:
Hi @lwintergerst, I researched more about this and was able to backup and delete some indices. Thank you!


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.