I have a question about translog. So here is the situation:
I have one index with 1 primary and 1 replica shard and continuously ingesting data. If i read documentation, it says that primary and replica shards are write their operations to the translog. Cmiiw
So my question is, if there is a issue for example the cluster become yellow or red, and the primary and replica shards become unassigned, how do they recover from the translog until they have the latest data? while the translog itself is written by the shard as written in the documentation
"so each shard copy also writes operations into its transaction log known as the translog"
if the shards becomes unassigned, then it can't receive the latest data right? therefore it recovers through translog. however, the translog itself is written by the shard. so even recovering via translog can't make the shard have the latest data. if so, how will the shard catch up with its data?
If a replica shard becomes unavailable, the primary shard will continue to accept writes and write these to the transaction log.
If the primary becomes unavailable and there is a replica shard in place, this will get promoted to primary and continue accepting writes as in the first scenario.
If the primary as well as all replicas of a specific shard becomes unavailable the index will go red and no writes will be accepeted to any shards in that index. When one of the unavailable shards become available this will become the primary and the index will again accept writes.
Ok, pretty clear. But i want to know one more thing. If the primary shard becomes unavailable, the replica shard become primary and accept writes. So my question is, the old primary shard, recover from which translog? because in the documentation on my first post, it said each shard, (which is primary and replica), has it's own translog. so will it recover using its own translog or using the replica's translog which obviously has the latest data?
If the old primary shard comes back it will become a replica shard (assuming a replica shard still need to be assigned) and recover from the current primary shard.
that means the translog is only used for reassigning the shards right? After that, the new replica will replicate the data from the current primary shard?
The transaction log is a local buffer holding indexd data until a new segment can be created during a flush. Writes are sent to both primary and secondary shards and are written to each shards transaction log.
In more recent versions writes use sequence numbers, which can be used when recovery takes place and to identify what has and has not been written to segments or local transaction log.
When the failed primary shard comes back in the example we discussed, the cluster can identify it is behind the current primary shard based on the sequence numbers. It will then recover based on the current primary shard. This may include procesing the local transaction log and then receive missed writes based on sequence numbers or simply copy the whole primary shard if it is very far behind.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.