Our index design :
The alias "my_alias" is created for the two indexes "myindex_a_2016_0106" and "myindex_a_2016_0712"
Insert data : Only insert data into the index "myindex_a_2016_0106" in the first half year in 2016, only insert data into the index "myindex_a_2016_0712" in the second half year in 2016, in future , new indexes such as myindex_a_2017_**** will be created to insert data.
Query data : Query by the alias "my_alias" instead of any specific index.
Our pain :
As the ES indexes data increases, we have a longer and longer backup time, each backup will take about 30min , the backup schedule is hourly in order to keep the data sync between production env and disaster recovery env. So ES almost have been backing up in 50% time. What's more , we will have more indexes in future. The backup time will be more longer .
Possible solution :
I am considering the approach to decrease the backup time. Currently we back up all indexes, Is it feasible to only back up the current index "myindex_a_2016_0712", which is the only index to insert data. My understanding is the files under "myindex_a_2016_0106" should NOT be updated in the second half year since I have NOT inserted any data into it , But It was really be updated, see the following pastes. Some of files were updated in Aug and Oct!! Why were they updated in the second half year ? Because of the query by alias ?
How to decrease the backup time in my case ?
If only backup the current index , will it largely decrease the backup time in future ? My concern are there is updating by ES itself in "myindex_a_2016_0106" but we do not backup it. Will there be any data lost in disaster recovery env after restoring the snapshots? What's more, we will not restore every snapshot , just restore the latest one in a regular schedule.
Besides my above idea , Any recommend from your side to decrease the backup duration ?
=====my indexes
[][root@esnode1 indices]$ ls
myindex_a_2016_0106 myindex_a_2016_0712
[][root@esnode1 indices]$ cd /{elasticsearch_data_dir}/nodes/0/indices/myindex_a_2016_0106/0/index
[][root@esnode1 indices]$ ll
-rw-r--r-- 1 root root 449 Aug 4 22:38 _bqu.si
-rw-r--r-- 1 root root 350 Aug 4 22:38 _bqv.cfe
......
-rw-r--r-- 1 root root 262 Aug 4 22:38 _bqz.si
-rw-r--r-- 1 root root 516 Oct 27 06:17 segments_2fh
-rw-r--r-- 1 root root 36 Oct 27 06:17 segments.gen
-rw-r--r-- 1 root root 0 Mar 8 2016 write.lock
[][root@esnode1 indices]$ date
Thu Oct 27 07:57:38 GMT 2016
With indices covering 6 months, you have up to 6 months worth of data being affected by merging, which results in newly created or merged segments needing to be included in the snapshot. If you instead used monthly indices, all the data that is over a month old should static and not need to be included in the snapshot repeatedly. How large are your indices/shards?
Each 6 months generates about 20G indices data , total about 45G indices now , shards number is 12
We ever tried to have more number of indices , but performance team said the performance was low with many indices after testing , so we switched to create a new index every 6 months .
If creating one index each month , then the indexes number will increase , how about the performance ?
From your point , it seems like it is feasible to just backup current index , but why was the last half year index files updated ? Will there be data lost if just back up current index ?
Snapshots work at the segment level, as described in this blog post. Indexing into an index causes new segments to be created and merged with older ones, which means that all data in that index will be snapshotted repeatedly as the data moves into new, larger segments. Once an index is no longer written to, no more merging should occur and no new segments that need to be snapshotted will be created.
From a query perspective having one index with 12 shards is roughly equivalent to having 12 indices with 1 shard. I therefore do not think performance will suffer much if you start using monthly indices with just 1 or 2 shards each. Given your data size 1 shard is probably sufficient, but you may want to increase that to 2 or 3 in order to be able to better spread out the data across your cluster, although that depends on your cluster size.
I am still not understand one point : I do not think the files under "myindex_a_2016_0106" should be
updated in the second half year since I have NOT inserted any data into it , But some files in it were really be updated in Aug and Oct!! Why were they updated in the second half year ? Because of the query by alias ?
Elasticsearch manages merging of segments automatically, and this will create new segments that will need to be backed up in a snapshot. If no data in the index has been added, updated or deleted I am not sure why it would do so after a reasonably long period of inactivity. Querying, directly or through an alias, should as far as I know not impact this.
"Elasticsearch manages merging of segments automatically, and this will
create new segments that will need to be backed up in a snapshot. "
----Q : Where are the new segments placed ? backup repository or index directory ?
Not sure my understanding is right , do you mean there should NOT be files updated in "myindex_a_2016_0106" indexes in my case ?
Elasticsearch stores data in segments, which make up shards, so they exist in Elasticsearch. It is at this level snapshot and restore operates, as described in the blog post I referred to earlier. If you are not indexing, updating or deleting any documents from the index, segments files should not be updated unless Elasticsearch decides to merge segments. Are you sure no modifications to the index are taking place? Are you tracking these metrics?
Will the backup action on the index "myindex_a_2016_0106" triger merge happening ?
If yes , the new files are from the backup action since I backup all indexes each time;
if no , there may be defect in our insert service
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
