some comments:
I write a script to get increase index per second, and below is the result.
What made me confused is why the increment of index is smooth? Is that
because of some feature of elasticsearch for the sake of parameters?
while i observed the delta index, the disk io of elasticsearch is very low
when the delta index is zero. For my opinion, I think the data didn't not
really hit elasticsearch. But, the flume(you can figure it out on my origin
post) which measure the data flow speed told me that the data didn't go
through to elasticsearch. It is odd.
Thu Aug 8 10:34:59 2013 , 25819
Thu Aug 8 10:35:01 2013 , 36627
Thu Aug 8 10:35:03 2013 , 15181
Thu Aug 8 10:35:05 2013 , 33000
Thu Aug 8 10:35:07 2013 , 8358
Thu Aug 8 10:35:09 2013 , 8644
Thu Aug 8 10:35:11 2013 , 0
Thu Aug 8 10:35:13 2013 , 0
Thu Aug 8 10:35:16 2013 , 0
Thu Aug 8 10:35:18 2013 , 0
Thu Aug 8 10:35:20 2013 , 0
Thu Aug 8 10:35:22 2013 , 0
Thu Aug 8 10:35:24 2013 , 28884
Thu Aug 8 10:35:26 2013 , 36
Thu Aug 8 10:35:28 2013 , 0
Thu Aug 8 10:35:30 2013 , 0
Thu Aug 8 10:35:33 2013 , 0
Thu Aug 8 10:35:35 2013 , 0
Thu Aug 8 10:35:37 2013 , 0
Thu Aug 8 10:35:39 2013 , 0
Thu Aug 8 10:35:41 2013 , 41036
Thu Aug 8 10:35:43 2013 , 35360
Thu Aug 8 10:35:45 2013 , 25924
Thu Aug 8 10:35:47 2013 , 25589
Thu Aug 8 10:35:49 2013 , 17866
Thu Aug 8 10:35:51 2013 , 11310
Thu Aug 8 10:35:54 2013 , 8135
Thu Aug 8 10:35:56 2013 , 28000
Thu Aug 8 10:35:58 2013 , 19044
Thu Aug 8 10:36:00 2013 , 0
Thu Aug 8 10:36:02 2013 , 0
Thu Aug 8 10:36:04 2013 , 0
Thu Aug 8 10:36:06 2013 , 0
Thu Aug 8 10:36:08 2013 , 0
Thu Aug 8 10:36:10 2013 , 0
在 2013年8月8日星期四UTC+8上午1时22分19秒，姚仁捷写道：
hey guys,
I'm using flume(tool of apache to transfer big log data) to load data to
elasticsearch in java. The problem I met is that the flume sometimes didn't
move data to elasticsearch. For research, I try to measure the index
number per second by calculating the delta index_total of the
elasticsearch cluster by RESTful API.
Well, I found that the delta index_total equaled zero in one or two
minutes, and, so that the flume hold the data which should be given to
elasticsearch. After the stop time of elasticsearch, flume will handle
the delayed data.
A simple brief conclusion is, the stop time makes flume unstable.
My question is, is the stop time usual? is that cuased by merge
segments? or the Full GC?
For Full GC, I remembered that elasticsearch will stop the world while
full gc. And on the basis of this, we must make Xmx and Xms the same. But I
found my elasticsearch node DO Full GC in about one hour, and the node
works well. I'm not sure how can we do to elasticsearch except set Xms the
same value as Xmx
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.