Meaning of Red in health status


(Mohit Anchlia) #1

I am using head and BigDesk and after my load testing yesterday I see the
health turned into Red but both of them don't tell me why it's Red. Is the
log best place to look?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Itamar Syn-Hershko) #2

Its usually a simple math check that answers this question. How many shards
you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see the
health turned into Red but both of them don't tell me why it's Red. Is the
log best place to look?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mohit Anchlia) #3

I just started out new and doing load test to figure out number of shards,
number of nodes required for my workload. So at the moment I have 1 shard
and 1 replica. But in Bigdesk I don't see any issues with GC or otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see the
health turned into Red but both of them don't tell me why it's Red. Is the
log best place to look?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Itamar Syn-Hershko) #4

and how many nodes do you have in the cluster? 1 replica will require you
to have at least 2 nodes

On Tue, Nov 19, 2013 at 9:02 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I just started out new and doing load test to figure out number of shards,
number of nodes required for my workload. So at the moment I have 1 shard
and 1 replica. But in Bigdesk I don't see any issues with GC or otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see
the health turned into Red but both of them don't tell me why it's Red. Is
the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #5

Use the lifecycle command of the es2unix tool on your logs. Look for when
nodes dropped out of the cluster.

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I just started out new and doing load test to figure out number of shards,
number of nodes required for my workload. So at the moment I have 1 shard
and 1 replica. But in Bigdesk I don't see any issues with GC or otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see
the health turned into Red but both of them don't tell me why it's Red. Is
the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mohit Anchlia) #6

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic ivan@brusic.com wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for when
nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see
the health turned into Red but both of them don't tell me why it's Red. Is
the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Itamar Syn-Hershko) #7

Decrease the replica count to 0 and you're all set, no need to look at the
logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic ivan@brusic.com wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for when
nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <itamar@code972.com

wrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I am using head and BigDesk and after my load testing yesterday I see
the health turned into Red but both of them don't tell me why it's Red. Is
the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mohit Anchlia) #8

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Decrease the replica count to 0 and you're all set, no need to look at the
logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic ivan@brusic.com wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
itamar@code972.com> wrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <mohitanchlia@gmail.com

wrote:

I am using head and BigDesk and after my load testing yesterday I see
the health turned into Red but both of them don't tell me why it's Red. Is
the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #9

Because you cannot store a primary and the only replica of the same index
on the same node, otherwise it defeats the point of having a replica.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 20 November 2013 06:55, Mohit Anchlia mohitanchlia@gmail.com wrote:

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Decrease the replica count to 0 and you're all set, no need to look at
the logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic ivan@brusic.com wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia <mohitanchlia@gmail.com

wrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
itamar@code972.com> wrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:

I am using head and BigDesk and after my load testing yesterday I
see the health turned into Red but both of them don't tell me why it's Red.
Is the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mohit Anchlia) #10

It appears that issue was with the heap size. I set it to 8G now on 32G
machine. Surprisingly elasticsearch didn't crash even though it was
throwing:

[2013-11-19 08:27:40,153][WARN ][index.engine.robin ] [10.80.140.242]
[test][0] failed engine

java.lang.OutOfMemoryError: Java heap space

at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)

On Tue, Nov 19, 2013 at 11:55 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Decrease the replica count to 0 and you're all set, no need to look at
the logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic ivan@brusic.com wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia <mohitanchlia@gmail.com

wrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
itamar@code972.com> wrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:

I am using head and BigDesk and after my load testing yesterday I
see the health turned into Red but both of them don't tell me why it's Red.
Is the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kurt Hurtado) #11

A good rule of thumb is to set ES_HEAP_SIZE to half the total system memory
(with max of 30GB however), unless you're running lots of other services on
that machine... YMMV!
Thanks,
Kurt

On Tuesday, November 19, 2013 12:54:08 PM UTC-8, Mo wrote:

It appears that issue was with the heap size. I set it to 8G now on 32G
machine. Surprisingly elasticsearch didn't crash even though it was
throwing:

[2013-11-19 08:27:40,153][WARN ][index.engine.robin ] [10.80.140.242]
[test][0] failed engine

java.lang.OutOfMemoryError: Java heap space

at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)

On Tue, Nov 19, 2013 at 11:55 AM, Mohit Anchlia <mohita...@gmail.com<javascript:>

wrote:

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko <ita...@code972.com<javascript:>

wrote:

Decrease the replica count to 0 and you're all set, no need to look at
the logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>

wrote:

I always had 1 node and I was successfuly indexing documents until last
night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic <iv...@brusic.com<javascript:>

wrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia <mohita...@gmail.com<javascript:>

wrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
ita...@code972.com <javascript:>> wrote:

Its usually a simple math check that answers this question. How many
shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>

wrote:

I am using head and BigDesk and after my load testing yesterday I
see the health turned into Red but both of them don't tell me why it's Red.
Is the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Itamar Syn-Hershko) #12

This has nothing to do with the red cluster state, just why the cluster
stopped being responsive, mind

On Tue, Nov 19, 2013 at 11:16 PM, Kurt Hurtado kurtado@gmail.com wrote:

A good rule of thumb is to set ES_HEAP_SIZE to half the total system
memory (with max of 30GB however), unless you're running lots of other
services on that machine... YMMV!
Thanks,
Kurt

On Tuesday, November 19, 2013 12:54:08 PM UTC-8, Mo wrote:

It appears that issue was with the heap size. I set it to 8G now on 32G
machine. Surprisingly elasticsearch didn't crash even though it was
throwing:

[2013-11-19 08:27:40,153][WARN ][index.engine.robin ] [10.80.140.242]
[test][0] failed engine

java.lang.OutOfMemoryError: Java heap space

at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)

On Tue, Nov 19, 2013 at 11:55 AM, Mohit Anchlia mohita...@gmail.comwrote:

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko <ita...@code972.com

wrote:

Decrease the replica count to 0 and you're all set, no need to look at
the logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohita...@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until
last night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic iv...@brusic.comwrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia mohita...@gmail.comwrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
ita...@code972.com> wrote:

Its usually a simple math check that answers this question. How
many shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <mohita...@gmail.com

wrote:

I am using head and BigDesk and after my load testing yesterday I
see the health turned into Red but both of them don't tell me why it's Red.
Is the log best place to look?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mohit Anchlia) #13

Thanks! I understand.

On Tue, Nov 19, 2013 at 1:24 PM, Itamar Syn-Hershko itamar@code972.comwrote:

This has nothing to do with the red cluster state, just why the cluster
stopped being responsive, mind

On Tue, Nov 19, 2013 at 11:16 PM, Kurt Hurtado kurtado@gmail.com wrote:

A good rule of thumb is to set ES_HEAP_SIZE to half the total system
memory (with max of 30GB however), unless you're running lots of other
services on that machine... YMMV!
Thanks,
Kurt

On Tuesday, November 19, 2013 12:54:08 PM UTC-8, Mo wrote:

It appears that issue was with the heap size. I set it to 8G now on 32G
machine. Surprisingly elasticsearch didn't crash even though it was
throwing:

[2013-11-19 08:27:40,153][WARN ][index.engine.robin ] [10.80.140.242]
[test][0] failed engine

java.lang.OutOfMemoryError: Java heap space

at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)

On Tue, Nov 19, 2013 at 11:55 AM, Mohit Anchlia mohita...@gmail.comwrote:

But why? I didn't have any issues yesterday and at that point I had 6M+
docs with 1 shard + 1 replica.

On Tue, Nov 19, 2013 at 11:47 AM, Itamar Syn-Hershko <
ita...@code972.com> wrote:

Decrease the replica count to 0 and you're all set, no need to look at
the logs

On Tue, Nov 19, 2013 at 9:43 PM, Mohit Anchlia mohita...@gmail.comwrote:

I always had 1 node and I was successfuly indexing documents until
last night. I'll look at the logs next.

On Tue, Nov 19, 2013 at 11:18 AM, Ivan Brusic iv...@brusic.comwrote:

Use the lifecycle command of the es2unix tool on your logs. Look for
when nodes dropped out of the cluster.

https://github.com/elasticsearch/es2unix

--
Ivan

On Tue, Nov 19, 2013 at 11:02 AM, Mohit Anchlia <mohita...@gmail.com

wrote:

I just started out new and doing load test to figure out number of
shards, number of nodes required for my workload. So at the moment I have 1
shard and 1 replica. But in Bigdesk I don't see any issues with GC or
otherwise.

On Tue, Nov 19, 2013 at 10:47 AM, Itamar Syn-Hershko <
ita...@code972.com> wrote:

Its usually a simple math check that answers this question. How
many shards you have, how many replicas each and how many nodes?

Also, are there any shard allocation rules defined?

On Tue, Nov 19, 2013 at 8:40 PM, Mohit Anchlia <
mohita...@gmail.com> wrote:

I am using head and BigDesk and after my load testing yesterday I
see the health turned into Red but both of them don't tell me why it's Red.
Is the log best place to look?

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #14