Incorrect Stats - Index time ES 2.0


(SK) #1

Hi,

I am trying to index 4 million docs in ES 2.0.2. After the indexing when i look into the stats(/_nodes/stats) API, i get around 35 minutes as index time. The way I calculated is by calculating the difference of index_time_in_millis before and after crawl. But immediately after the execution of my program( which took 23 minutes), when i verified using cats api (_cat/indices?v) I could see 4 million docs
I am having 5 shards and 0 replicas running on a single node.

My question is how is that stats(/_nodes/stats) api showing 35 minutes but the actual time was around 23 minutes. I am seeing this in ES 2.0 onwards but in ES 1.4 both my program total execution time and ES stats time was same.

Stats Before start of Indexing

indexing: {
index_total: 7715578,
index_time_in_millis: 3634410,
index_current: 0,
delete_total: 5515022,
delete_time_in_millis: 794744,
delete_current: 0,
noop_update_total: 0,
is_throttled: false,
throttle_time_in_millis: 0

Stats After completion of Indexing

indexing: {
index_total: 12116702,
index_time_in_millis: 5735582,
index_current: 0,
delete_total: 5515022,
delete_time_in_millis: 794744,
delete_current: 0,
noop_update_total: 0,
is_throttled: false,
throttle_time_in_millis: 0

Indexing Time is (5735582-3634410)/60000 =35.0195 minutes


(SK) #2

Stats found when running on ES 1.4.4 same machine as ES2.0.2

After indexing

indexing: {
index_total: 4401126,
index_time_in_millis: 1762773,
index_current: 0,
delete_total: 2,
delete_time_in_millis: 5,
delete_current: 0,
noop_update_total: 0,
is_throttled: false,
throttle_time_in_millis: 0
}

Here the program took 29 mins and the index stats showed the same time too...


(SK) #3

@jprante Any thoughts on this


(system) #4