Thanks, again, Kurt,
I looked at _status output and I think what I found was inconsistency disappears over time.
I wrote a simple ruby script to process _status?pretty=true output (see below)
I took a couple of sample data and saw many shards have different translog id but the num_docs seem ok.
What does translog id actually mean?
What is your concern if replicas have different translog id?
One more question for anybody knows ES well:
Some document seemed to take hours until it appeared in the replica (or maybe viceversa).
What brings this time lag?
----------------------------------------- here's my script --------------
if (ARGV.length < 1)
puts $PROGRAM_NAME + ' '
exit
end
@mode = :waiting_index
@info = {}
open(ARGV[0], 'r') {|f|
f.each_line {|line|
case @mode
when :waiting_index
if (line =~ /"(i[0-9]+)" : /)
@index = $1
@info[@index] = {}
@mode = :waiting_shard_no
end
when :waiting_shard_no
if (line =~ /"([0-9]+)" : [/)
@shard = $1
@info[@index][@shard] = {}
@mode = :handling_shard
@subshard = 0
@info[@index][@shard][@subshard] = {}
end
when :handling_shard
if (line =~ /"primary" : (\w+),/)
@info[@index][@shard][@subshard][:primary] = $1
elsif (line =~ /"node" : "(\S+)",/)
@info[@index][@shard][@subshard][:node] = $1
elsif (line =~ /"id" : ([0-9]+)/)
@info[@index][@shard][@subshard][:translog_id] = $1
elsif (line =~ /"num_docs" : ([0-9]+)/)
@info[@index][@shard][@subshard][:num_docs] = $1
elsif (line =~ /^\s+}, {\s?$/)
@subshard += 1
@info[@index][@shard][@subshard] = {}
elsif (line =~ /^\s+} ],\s?$/)
@mode = :waiting_shard_no
elsif (line =~ /^\s+} ]\s?$/)
@mode = :waiting_index
end
end
}
}
def dump_info(info)
"tranalog=#{info[:translog_id]} num_docs=#{info[:num_docs]} node=#{info[:node]}" +
(info[:primary] == "true" ? " primary" : " ")
end
def dump_diff(idx, shard, i, info, ref_i, ref_info)
idxShard = "#{idx}-#{shard}"
"#{idxShard} [#{ref_i}]: #{dump_info(ref_info)}\n" +
" " * idxShard.length + " [#{i}]: #{dump_info(info)}"
end
def sort_keys(col)
col.keys.sort {|a,b|
if (a.class != String || a.length == b.length)
a <=> b
else
a.length <=> b.length
end
}
end
sort_keys(@info).each {|idx| idx_info = @info[idx]
sort_keys(idx_info).each {|shard| shard_info = idx_info[shard]
translogs = []
sort_keys(shard_info).each {|i| info = shard_info[i]
if (translogs.empty?)
translogs << { i => info }
else
if (info[:num_docs] != translogs[0].values[0][:num_docs])
puts dump_diff(idx, shard, i, info, translogs[0].keys[0], translogs[0].values[0]) + " DIFFERENT COUNT"
elsif (info[:translog_id] != translogs[0].values[0][:translog_id])
puts dump_diff(idx, shard, i, info, translogs[0].keys[0], translogs[0].values[0])
end
end
}
}
}