I've been implementing an ELK stack for the past year or so. I had thought
that we would have plenty of space, but recently added a log source that
increased the number of log entries a day by around 30x. That prompted me
to start looking into ways of managing ES's data storage in order to keep
from running out of space. Which led me to Curator and Snapshots.
If I am reading the documentation[1] for both systems correctly, I think I
can do the following:
Create a repository for old data.
Use a cron job and Curator to automatically take snapshots of data
older than a certain time period (say, 6 months).
Then have Curator delete the data older than that time period.
The result would be that all data older than the time period would
be stored in the repository. The data would be compressed (what kind of
compression?)
When I have need for data older than the time period, I could use
Curator to restore it to the ES cluster, or even a different ES cluster.
After that I could do what I needed, before deleting it again.
I'd test all this myself, but I don't have the resources for a decent test
environment yet. Still working on that.
Am I missing anything? Are there better ways to keep from running out of
storage space? Any general advice related to this kind of thing?
However, it should be noted that Curator is only for taking snapshots—it
cannot restore them. This functionality was omitted because restoring is
not typically a daily occurrence, like the other procedures Curator does.
Fortunately, restoring indices is a relatively simple thing to do with the
API (modules-snapshots.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore
).
--Aaron
On Wednesday, July 30, 2014 12:10:36 PM UTC-5, David Reagan wrote:
I've been implementing an ELK stack for the past year or so. I had thought
that we would have plenty of space, but recently added a log source that
increased the number of log entries a day by around 30x. That prompted me
to start looking into ways of managing ES's data storage in order to keep
from running out of space. Which led me to Curator and Snapshots.
If I am reading the documentation[1] for both systems correctly, I think I
can do the following:
Create a repository for old data.
Use a cron job and Curator to automatically take snapshots of data
older than a certain time period (say, 6 months).
Then have Curator delete the data older than that time period.
The result would be that all data older than the time period
would be stored in the repository. The data would be compressed (what kind
of compression?)
When I have need for data older than the time period, I could use
Curator to restore it to the ES cluster, or even a different ES cluster.
After that I could do what I needed, before deleting it again.
I'd test all this myself, but I don't have the resources for a decent test
environment yet. Still working on that.
Am I missing anything? Are there better ways to keep from running out of
storage space? Any general advice related to this kind of thing?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
Still working on that.