What actually causes Red / Yellow cluster health

Elastic Stack Elasticsearch
1

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I can
go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02120a2d-52cc-4a55-afae-d90e816522c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Yellow means unassigned replicas. Red means unassigned primaries.

It'll change (quickly) if you add a new index, as it needs to create and
allocate primaries and then secondaries.

On 3 March 2015 at 21:49, Steven Williams cybordax@googlemail.com wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I can
go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02120a2d-52cc-4a55-afae-d90e816522c8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02120a2d-52cc-4a55-afae-d90e816522c8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X92wLCFbNRVb5h6CS4y%3D2%2B0Rj14CAr8LjBNMayv8Wjdyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hello - thanks for the reply Mark, greatly appreciated.

i get what the different codes mean.., but not why they actually happen.

the two nodes I have.
one of them runs scheduled jobs that gets data from a products table from
mySql into elastic. it does this every day at 8am as a master update.
then it updates stock for these items, every 15 mins.

it does this every day and every 15 mins, and currently again, the status
is green for my cluster.
so why would it suddenly go to yellow if the nodes are doing what they
normally do on a daily basis?

im trying to identify if I have a network issue, a cpu load issue, a memory
leak issue etc. so then I can address them and fix it, so that I dont get
yellow status again.

Cheers

On Tuesday, 3 March 2015 11:47:52 UTC, Mark Walkom wrote:

Yellow means unassigned replicas. Red means unassigned primaries.

It'll change (quickly) if you add a new index, as it needs to create and
allocate primaries and then secondaries.

On 3 March 2015 at 21:49, Steven Williams <cybo...@googlemail.com
<javascript:>> wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02120a2d-52cc-4a55-afae-d90e816522c8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02120a2d-52cc-4a55-afae-d90e816522c8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0c919cd-5bb6-4d79-bdad-c39436e0d8eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

What sort of monitoring are you having.. and how are you getting alerts
when the state of the cluster changes..? Is there some plugin that I can
install?

We are facing similar problems as well.. Even more scary my nodes are
leaving the cluster from time to time and rejoin the cluster.. Because of
this lots of shards go to "unassigned" state and it takes several minutes
for the cluster to get back to normal green state..

I came across this blog post that suggested me to change my garbage
collector to G1GC from the default ConcMarkSweepGC, so that stop the world
garbage collections will not occur anymore....
Still my nodes are leaving the cluster :open_mouth:

Can you tell me what sort of monitoring are you using to get alerts when
cluster state changes.

Thank you.

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I can
go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a8f80f04-b8ba-429b-be7e-751564b76f6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Link to blog
post http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

On Tuesday, March 3, 2015 at 6:03:10 PM UTC+5:30, Anil Karaka wrote:

What sort of monitoring are you having.. and how are you getting alerts
when the state of the cluster changes..? Is there some plugin that I can
install?

We are facing similar problems as well.. Even more scary my nodes are
leaving the cluster from time to time and rejoin the cluster.. Because of
this lots of shards go to "unassigned" state and it takes several minutes
for the cluster to get back to normal green state..

I came across this blog post that suggested me to change my garbage
collector to G1GC from the default ConcMarkSweepGC, so that stop the world
garbage collections will not occur anymore....
Still my nodes are leaving the cluster :open_mouth:

Can you tell me what sort of monitoring are you using to get alerts when
cluster state changes.

Thank you.

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09439944-fd53-4652-8d1c-5a374d10c05a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Hello Anil - thanks for your reply

Im glad that someone else is experiencing this too!

Ive made my own custom monitor.. using vb.net

  • its simply an application that runs every so many minutes (currently 5)
    via scheduled task on windos machine that looks at the health of the
    cluster..
    so it would look at the ipaddress where the elastic db is then the port
    number..

so for example http://192.169.1.1:9200
then the api that looks for the health.. so the entire url would be.

http://192.169.1.1:9200/_cluster/health?pretty=true

and I get the below result...

{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

.. then the bit that says status green..

I look for that in my app and if its anything else other than green, I make the app email myself.

This works for me, BUT.. it still does not explain WHY these things happen, and I would rather diagnose why these happen,to stop them from happening again.

I dont have nodes leaving my cluster..

I have two separate machines running windows, and they are both always in the cluster, but for whatever reason - the cluster sees fit to turn yellow on occasion, (not regularly, but every now and then) which I understand that replicas didnt happen on some shards.. but I want to know whyyyyyy lol :slight_smile:

Cheers :slight_smile:

On Tuesday, 3 March 2015 12:33:10 UTC, Anil Karaka wrote:

What sort of monitoring are you having.. and how are you getting alerts
when the state of the cluster changes..? Is there some plugin that I can
install?

We are facing similar problems as well.. Even more scary my nodes are
leaving the cluster from time to time and rejoin the cluster.. Because of
this lots of shards go to "unassigned" state and it takes several minutes
for the cluster to get back to normal green state..

I came across this blog post that suggested me to change my garbage
collector to G1GC from the default ConcMarkSweepGC, so that stop the world
garbage collections will not occur anymore....
Still my nodes are leaving the cluster :open_mouth:

Can you tell me what sort of monitoring are you using to get alerts when
cluster state changes.

Thank you.

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b34b57eb-427a-4e6d-b536-91cad205f438%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I can
go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9960c296-9e74-476a-8f3c-9341e43a6d90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically considered
the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock levels
    that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take parameters
so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/65bd6280-c761-42e6-8d66-36a5e081691d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can rdp
into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock levels
    that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d05db214-f7f0-4721-81de-609648625977%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and then
if NOT green, it will email you on the address you specify from the smtp
host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio 2010, i
can give you the sourcecode / project so if you wished you could alter the
program further (and also check the code out to see that it is not doing
anything malicious) :slight_smile:

On Tue, Mar 3, 2015 at 1:09 PM, Anil Karaka anilkaraka@gmail.com wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can rdp
into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock levels
    that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/m5QNsmh9Mek/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d05db214-f7f0-4721-81de-609648625977%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d05db214-f7f0-4721-81de-609648625977%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANRN3tb3wESb3UA46Tas%2B2WskUopMo0mwY8teLB9hwWm-vUuwA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

11

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and then
if NOT green, it will email you on the address you specify from the smtp
host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio 2010, i
can give you the sourcecode / project so if you wished you could alter the
program further (and also check the code out to see that it is not doing
anything malicious) :slight_smile:

On Tuesday, 3 March 2015 13:09:25 UTC, Anil Karaka wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can rdp
into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock levels
    that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f02c6470-a56a-458c-bc49-7ed46d41f1af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

12

I think you I should write a simple python script myself..

I don't think I will be installing visual studio in that windows machine..

Thanks for your help though :slight_smile:

On Tuesday, March 3, 2015 at 6:46:21 PM UTC+5:30, Steven Williams wrote:

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and then
if NOT green, it will email you on the address you specify from the smtp
host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio 2010, i
can give you the sourcecode / project so if you wished you could alter the
program further (and also check the code out to see that it is not doing
anything malicious) :slight_smile:

On Tuesday, 3 March 2015 13:09:25 UTC, Anil Karaka wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can
rdp into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock
    levels that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5084541f-0e4d-41ea-86b5-123240059353%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

13

your machine does not need visual studio on :slight_smile:

i just said that in case you wanted to look at the code for any reason, and
have the ability to alter it.

the app will still work, so long as it has the latest dotnet framework and
also can see the elastic server.

do you still want me to adapt my app for you to try on your server?

On Tuesday, 3 March 2015 13:29:15 UTC, Anil Karaka wrote:

I think you I should write a simple python script myself..

I don't think I will be installing visual studio in that windows machine..

Thanks for your help though :slight_smile:

On Tuesday, March 3, 2015 at 6:46:21 PM UTC+5:30, Steven Williams wrote:

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and then
if NOT green, it will email you on the address you specify from the smtp
host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio 2010,
i can give you the sourcecode / project so if you wished you could alter
the program further (and also check the code out to see that it is not
doing anything malicious) :slight_smile:

On Tuesday, 3 March 2015 13:09:25 UTC, Anil Karaka wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can
rdp into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock
    levels that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams
wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and
I can go and restart the slave server, then it goes back to green.. but
still this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ab83dd45-9fc5-4a70-bf92-2625b2623376%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

14

Thanks.. will give it a try..

fileds to specify my master node. and email address..?

On Tuesday, March 3, 2015 at 7:04:15 PM UTC+5:30, Steven Williams wrote:

your machine does not need visual studio on :slight_smile:

i just said that in case you wanted to look at the code for any reason,
and have the ability to alter it.

the app will still work, so long as it has the latest dotnet framework
and also can see the elastic server.

do you still want me to adapt my app for you to try on your server?

On Tuesday, 3 March 2015 13:29:15 UTC, Anil Karaka wrote:

I think you I should write a simple python script myself..

I don't think I will be installing visual studio in that windows machine..

Thanks for your help though :slight_smile:

On Tuesday, March 3, 2015 at 6:46:21 PM UTC+5:30, Steven Williams wrote:

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and
then if NOT green, it will email you on the address you specify from the
smtp host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio 2010,
i can give you the sourcecode / project so if you wished you could alter
the program further (and also check the code out to see that it is not
doing anything malicious) :slight_smile:

On Tuesday, 3 March 2015 13:09:25 UTC, Anil Karaka wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can
rdp into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is that
enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock
    levels that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams
wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow
health?

ive got monitoring on my cluster to alert me when its not green, and
I can go and restart the slave server, then it goes back to green.. but
still this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f93e0c5a-6d6c-42c8-98ae-02bdc3dfc737%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

15

Hello

so, just altered the app on my server, and it works ok.

when you set up the scheduled task, or even the shortcut to the exe to try
it..

the parameters need to be separated by space..

so example

ElasticHealthMonitor.exe
http://192.168.2.2:9200/_cluster/health?pretty=true anil@karaka.com
mail.yourmailserver.com

or the IP address of your mail server, whichever works.

then you will get an email come back to you if all is ok, and if all is not
ok..

I can zip up the exe and send to you via your gmail account - that ok?

On Tuesday, 3 March 2015 13:51:49 UTC, Anil Karaka wrote:

Thanks.. will give it a try..

fileds to specify my master node. and email address..?

On Tuesday, March 3, 2015 at 7:04:15 PM UTC+5:30, Steven Williams wrote:

your machine does not need visual studio on :slight_smile:

i just said that in case you wanted to look at the code for any reason,
and have the ability to alter it.

the app will still work, so long as it has the latest dotnet framework
and also can see the elastic server.

do you still want me to adapt my app for you to try on your server?

On Tuesday, 3 March 2015 13:29:15 UTC, Anil Karaka wrote:

I think you I should write a simple python script myself..

I don't think I will be installing visual studio in that windows
machine..

Thanks for your help though :slight_smile:

On Tuesday, March 3, 2015 at 6:46:21 PM UTC+5:30, Steven Williams wrote:

yeah that will work, so long as the windows server has the latest .net
framework on

then all it will do is simply make a http request to the URL that you
specify when calling the app,

it will then get the JSON response from your Elastic Custer URL, and
then if NOT green, it will email you on the address you specify from the
smtp host you specify :slight_smile:

Want to give it a go?

ill need to tweak the app my end... and if you have visual studio
2010, i can give you the sourcecode / project so if you wished you could
alter the program further (and also check the code out to see that it is
not doing anything malicious) :slight_smile:

On Tuesday, 3 March 2015 13:09:25 UTC, Anil Karaka wrote:

My Elasticsearch cluster is is on Ubuntu..

but I also have a small windows instance inside the VPC, so that I can
rdp into it to use plugins like kopf, bigdesk etc..

Basically I have a windows instance that can query my cluster.. Is
that enough to use your solution?

Thanks a bunch :slight_smile:

On Tuesday, March 3, 2015 at 6:31:11 PM UTC+5:30, Steven Williams
wrote:

Hello

Stays in Green status: for about a couple of weeks
Goes to Yellow Status: until I reboot the node that is basically
considered the slave machine.

Rivers: 1
Index/tables: 2
one table is the Kibana thing I installed to try it out.
the other table is the main products table that has about 1.8 million
records.

  • main master update done on it every day at 8am
  • smaller stock level updates done every 15 minutes only for stock
    levels that have changed since 15 mins ago.

Do you want me to see if I can adapt my app so that it will take
parameters so you can use it on your system?
needs to be on a windows machine though :slight_smile:

On Tuesday, 3 March 2015 12:53:58 UTC, Anil Karaka wrote:

How long does your cluster stay in that state?

and how many indexes in your cluster?

replica/shard count etc etc?

On Tuesday, March 3, 2015 at 4:19:13 PM UTC+5:30, Steven Williams
wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the
cluster goes from green to yellow (my cluster health is below, and at the
moment all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow
health?

ive got monitoring on my cluster to alert me when its not green,
and I can go and restart the slave server, then it goes back to green.. but
still this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dd5dfc92-126b-4d7a-98ab-f188e7c3c9d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

16

just sent an email to your gmail account can you confirm you have
received it

On Tuesday, 3 March 2015 10:49:13 UTC, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I can
go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e5861f79-f0fd-4c08-90d0-5b8d50271b30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

17

Yep I got your email..

On Tuesday, March 3, 2015 at 7:35:56 PM UTC+5:30, Steven Williams wrote:

just sent an email to your gmail account can you confirm you have
received it

On Tuesday, 3 March 2015 10:49:13 UTC, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster goes
from green to yellow (my cluster health is below, and at the moment all is
good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8714358b-cfe3-4755-bc19-8d42fdde9697%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

18

ive resent it - as googlemail didnt like the fact an exe was in a zip file.
so renamed the exe to ._exe :slight_smile: so just rename back to .exe etc

On Tuesday, 3 March 2015 14:10:00 UTC, Anil Karaka wrote:

Yep I got your email..

On Tuesday, March 3, 2015 at 7:35:56 PM UTC+5:30, Steven Williams wrote:

just sent an email to your gmail account can you confirm you have
received it

On Tuesday, 3 March 2015 10:49:13 UTC, Steven Williams wrote:

Hello all

I have 2 nodes (two different machines) and on occasion the cluster
goes from green to yellow (my cluster health is below, and at the moment
all is good)

"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

I have read what the different status colours mean, but I cant find
anywhere that says WHY these things happen i.e. network issue, cpu load etc
etc.

Are the any main reasons that would give cause to red / yellow health?

ive got monitoring on my cluster to alert me when its not green, and I
can go and restart the slave server, then it goes back to green.. but still
this does not identify the root cause as to why this happens?

Any comments would be greatly appreciated :slight_smile:

Cheers

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f68602d-ac26-4cf3-b4d6-f7fd487ea816%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.