Cached Elasticsearch Information

I am new to ELK and I am still using a dev environment with real data to
understand how the pipeline works.

Recently I had nodes networking between them that were part of a 4 node
cluster: 1 master node, no data and 3 data-only nodes. These were working
fine up to the point that they lost connectivity between themselves. I am
using a unicast cluster setup as multicast was not working on my OpenStack
cluster (a question for another future post).

When I rebooted the nodes and tried to bring the master back up it tried to
join the previous master instance. Clearly there is information cached
about the previous running instance that needs to be flushed. As this is a
dev environment, I copied the config file and blew away the elastic search
directory, copied the config back and tried to restart elasticsearch.
Curiously it is still trying to join the old master, even though no other
processes are running. Obviously this cached data lives somewhere else on
the system and I have yet to find it.

Perhaps someone can point me in the right direction here.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It doesn't cache this sort of info, it'll read what is in the config file.

Are you using puppet/chef/other for config management perchance? These
could be over writing your config.

On 15 January 2015 at 06:22, Albion Baucom albionb@gene.com wrote:

I am new to ELK and I am still using a dev environment with real data to
understand how the pipeline works.

Recently I had nodes networking between them that were part of a 4 node
cluster: 1 master node, no data and 3 data-only nodes. These were working
fine up to the point that they lost connectivity between themselves. I am
using a unicast cluster setup as multicast was not working on my OpenStack
cluster (a question for another future post).

When I rebooted the nodes and tried to bring the master back up it tried
to join the previous master instance. Clearly there is information cached
about the previous running instance that needs to be flushed. As this is a
dev environment, I copied the config file and blew away the elastic search
directory, copied the config back and tried to restart elasticsearch.
Curiously it is still trying to join the old master, even though no other
processes are running. Obviously this cached data lives somewhere else on
the system and I have yet to find it.

Perhaps someone can point me in the right direction here.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X96sBGgJiEqfWc6X9GTHHvfUryCOSpSUS%3DqFj8_HMcdtA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

No configuration management in place. I manually untarred the files. So all
configuration information should be contained in the application directory?

Here is what I see which is, even after a reboot, a Nightwatch named
Elasticsearch instance looking for the phantom "Plantman" master, even
though this node is configured to be a master only:

[root@es-master elasticsearch-1.4.2]#
[2015-01-16 11:57:43,500][INFO ][node ] [Nightwatch]
version[1.4.2], pid[24139], build[927caff/2014-12-16T14:11:12Z]
[2015-01-16 11:57:43,501][INFO ][node ] [Nightwatch]
initializing ...
[2015-01-16 11:57:43,505][INFO ][plugins ] [Nightwatch]
loaded [], sites []
[2015-01-16 11:57:46,701][INFO ][node ] [Nightwatch]
initialized
[2015-01-16 11:57:46,701][INFO ][node ] [Nightwatch]
starting ...
[2015-01-16 11:57:46,903][INFO ][transport ] [Nightwatch]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.0.0.45:9300]}
[2015-01-16 11:57:46,919][INFO ][discovery ] [Nightwatch]
lsflogs/A6CISFuzRkK31rL2dFFdNA
[2015-01-16 11:57:50,187][INFO ][discovery.zen ] [Nightwatch] failed
to send join request to master [[Plantman]
[VOmbX4yORHiObk7Q3D7tbQ][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}], reason
[RemoteTransportException[[Nightwatch][inet[/10.0.0.45:9300]][internal:discovery/zen/join]];
nested: ElasticsearchIllegalStateException[Node
[[Nightwatch][A6CISFuzRkK31rL2dFFdNA][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}] not master for join request from
[[Nightwatch][A6CISFuzRkK31rL2dFFdNA][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}]]; ], tried [3] times

My workaround was to rename the cluster. Perhaps there was another node
with that information and it was confusing the master?

But you answered my question which is there is no information written
outside of the application directory that could cache settings for future
clusters if I tear them down and rebuild them.

Thanks
Albion

On Wednesday, January 14, 2015 at 6:42:23 PM UTC-8, Mark Walkom wrote:

It doesn't cache this sort of info, it'll read what is in the config file.

Are you using puppet/chef/other for config management perchance? These
could be over writing your config.

On 15 January 2015 at 06:22, Albion Baucom <alb...@gene.com <javascript:>>
wrote:

I am new to ELK and I am still using a dev environment with real data to
understand how the pipeline works.

Recently I had nodes networking between them that were part of a 4 node
cluster: 1 master node, no data and 3 data-only nodes. These were working
fine up to the point that they lost connectivity between themselves. I am
using a unicast cluster setup as multicast was not working on my OpenStack
cluster (a question for another future post).

When I rebooted the nodes and tried to bring the master back up it tried
to join the previous master instance. Clearly there is information cached
about the previous running instance that needs to be flushed. As this is a
dev environment, I copied the config file and blew away the elastic search
directory, copied the config back and tried to restart elasticsearch.
Curiously it is still trying to join the old master, even though no other
processes are running. Obviously this cached data lives somewhere else on
the system and I have yet to find it.

Perhaps someone can point me in the right direction here.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/12876abb-4c7c-4ef8-934f-d0afac8b8c3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

If you are just untarring and starting the ES binary, it'll use the
defaults, which is multicast and default cluster name.
So it'll do a search for any other nodes and if they have the same cluster
name and can reach each other on the network, they will try to form a
cluster, that is what you are seeing.

On 17 January 2015 at 09:15, Albion Baucom albionb@gene.com wrote:

No configuration management in place. I manually untarred the files. So
all configuration information should be contained in the application
directory?

Here is what I see which is, even after a reboot, a Nightwatch named
Elasticsearch instance looking for the phantom "Plantman" master, even
though this node is configured to be a master only:

[root@es-master elasticsearch-1.4.2]#
[2015-01-16 11:57:43,500][INFO ][node ] [Nightwatch]
version[1.4.2], pid[24139], build[927caff/2014-12-16T14:11:12Z]
[2015-01-16 11:57:43,501][INFO ][node ] [Nightwatch]
initializing ...
[2015-01-16 11:57:43,505][INFO ][plugins ] [Nightwatch]
loaded [], sites []
[2015-01-16 11:57:46,701][INFO ][node ] [Nightwatch]
initialized
[2015-01-16 11:57:46,701][INFO ][node ] [Nightwatch]
starting ...
[2015-01-16 11:57:46,903][INFO ][transport ] [Nightwatch]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.0.0.45:9300]}
[2015-01-16 11:57:46,919][INFO ][discovery ] [Nightwatch]
lsflogs/A6CISFuzRkK31rL2dFFdNA
[2015-01-16 11:57:50,187][INFO ][discovery.zen ] [Nightwatch] failed
to send join request to master [[Plantman]

[VOmbX4yORHiObk7Q3D7tbQ][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}], reason
[RemoteTransportException[[Nightwatch][inet[/10.0.0.45:9300]][internal:discovery/zen/join]];
nested: ElasticsearchIllegalStateException[Node
[[Nightwatch][A6CISFuzRkK31rL2dFFdNA][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}] not master for join request from
[[Nightwatch][A6CISFuzRkK31rL2dFFdNA][es-master][inet[/10.0.0.45:9300]]{data=false,
master=true}]]; ], tried [3] times

My workaround was to rename the cluster. Perhaps there was another node
with that information and it was confusing the master?

But you answered my question which is there is no information written
outside of the application directory that could cache settings for future
clusters if I tear them down and rebuild them.

Thanks
Albion

On Wednesday, January 14, 2015 at 6:42:23 PM UTC-8, Mark Walkom wrote:

It doesn't cache this sort of info, it'll read what is in the config file.

Are you using puppet/chef/other for config management perchance? These
could be over writing your config.

On 15 January 2015 at 06:22, Albion Baucom alb...@gene.com wrote:

I am new to ELK and I am still using a dev environment with real data to
understand how the pipeline works.

Recently I had nodes networking between them that were part of a 4 node
cluster: 1 master node, no data and 3 data-only nodes. These were working
fine up to the point that they lost connectivity between themselves. I am
using a unicast cluster setup as multicast was not working on my OpenStack
cluster (a question for another future post).

When I rebooted the nodes and tried to bring the master back up it tried
to join the previous master instance. Clearly there is information cached
about the previous running instance that needs to be flushed. As this is a
dev environment, I copied the config file and blew away the elastic search
directory, copied the config back and tried to restart elasticsearch.
Curiously it is still trying to join the old master, even though no other
processes are running. Obviously this cached data lives somewhere else on
the system and I have yet to find it.

Perhaps someone can point me in the right direction here.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/90187018-dde9-447d-b211-d806fd46cec8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/12876abb-4c7c-4ef8-934f-d0afac8b8c3b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12876abb-4c7c-4ef8-934f-d0afac8b8c3b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7Hn2ka9ExL27YE5Z6X96L0_jN7pDpA6U0P%2BO%2BM860fA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.