Hi,
Let me give a brief description of my cluster.
Today, we have 10 nodes (10 servers), each node with 30GB RAM, 400GB SSD
Raid 1, 8 cores. The ES Cluster has 10 indexes which are divided between
the nodes. Each index has 10 shards, only 2 (smaller indexes) have replica.
All indexes have a total of 690GB which are divided between nodes, getting
69GB per node. Only onde index has 530.9gb and is responsible for almost
90% of the size.
The problem occurred again today.
As you recommended, after high IO, I checked if it had any index with some
fragment size generated recently. I can guarantee that there was no merge
involving large fragments. See ( http://i.imgur.com/we1rAix.png )
On newrelic, I can see that the IO is rising over time. There are four days
since I restarted my cluster and the IO is growing. See (
http://i.imgur.com/u0EQdwv.png)
When node "esdoc6" had the problem, i ran iotop command to check if it was
really only ES consuming all IO (330mb/s). See (
http://i.imgur.com/c1tCnM7.png)
I'll try to upgrade to the newer version of ES that just came out.
Any questions or suggestions will be cool.
I'll post any news on the status of the cluster
2013/8/7 simonw simon.willnauer@elasticsearch.com
hey,
I'd guess that you are seeing the impact of segment merges. I am wondering
if you see a drop in index size (physical size on disk) on the nodes once
they come back to "normal". I am curious what causes this but I guess it's
merging. Do you have a lot of update coming in and can you elaborate about
your setup a little? From the screenshot I can tell that you have 20M docs
but it seems to be on a single node? I can't really make sense out of the
screenshot, can you provide some more output from your cluster?
simon
On Wednesday, August 7, 2013 2:52:38 PM UTC+2, Gustavo Maia wrote:
Hi, Simonw,
Sorry for the delay.
I'm not using any kind of route and also not changed anything in my
index, just deployed the new version of the ES.
I configured cron to be able to run the optimizer every day at 00:00 when
my traffic is lower. With this change I noticed an improvement. The number
of concurrent query is normal in all nodes, however IO starts to increase
with time. When I restart the cluster IO is 30% and over time (1, 2 days)
shall be 60% (max)
Any questions or suggestions will be cool.
I'll post any news on the status of the cluster
Gustavo Maia
2013/8/4 simonw <simon.w...@**elasticsearch.com>
hey,
are you using any kind of routing? did you change anything in the way
you query your data after you upgraded?
simon
On Thursday, August 1, 2013 7:33:16 PM UTC+2, Gustavo Maia wrote:
I have some problems in my ES cluster when i upgraded from version
0.20.1 to version 0.90.1.
My cluster has 10 servers. At least once a day one node has I/O 100%,
looking at plugin paramedic i saw that an node gets 50-60 concurrent
queries and others get 1 or 2 queries.
When i restart the cluster that node back to normal. But after 30min
40min another node goes to I / O 100%.
When I execute full restart, my cluster keeps for a longer time
without the problem occurs.
I'm adding the screen shout of the paramedic plugin to see if it helps.
http://i.imgur.com/utOR3jJ.png
Thanks
--
Gustavo Maia
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.
--
Gustavo Maia
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Gustavo Maia
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
