Gateway access required on non-data client nodes on startup if its the first node up?


(cwho) #1

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding data
and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available -
as if they were not created, guessing the metadata isn't read from the
gateway.

I cannot guarantee order of startup if there's a network/power outage.
I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(Shay Banon) #2

Can you try and use 0.14.2, see if it helps? Also, note that you get long
term persistency with the default local gateway as well. So make sure you
want to use the fs gateway.

On Wed, Jan 5, 2011 at 4:44 PM, cwho80 fuzzybr80@gmail.com wrote:

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding data
and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available -
as if they were not created, guessing the metadata isn't read from the
gateway.

I cannot guarantee order of startup if there's a network/power outage.
I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(cwho) #3

In 0.14.2 the metadata is now being saved on the client non-data node
too. However ClusterHealthRequest for a specific index seems to be not
working as expected now....

I run with an index named "test".
When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

in 0.13.1, if its not connected - check returns RED or index not found
exception - which was great :).

in 0.14.2, if its not connected - it returns GREEN initially
consistently even if the index is not there yet - breaking the check.
Subsequently I tried indexing something. It then created the index
(metadata appears in data directory on the client node), then threw an
exception ( failed to index - expected because there is no data node)

  • and check subsequently returns RED.

On Jan 6, 4:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you try and use 0.14.2, see if it helps? Also, note that you get long
term persistency with the default local gateway as well. So make sure you
want to use the fs gateway.

On Wed, Jan 5, 2011 at 4:44 PM, cwho80 fuzzyb...@gmail.com wrote:

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding data
and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available -
as if they were not created, guessing the metadata isn't read from the
gateway.

I cannot guarantee order of startup if there's a network/power outage.
I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(cwho) #4

When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

I meant:
When my client node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

sorry for the confusion.

On Jan 7, 10:12 am, cwho80 fuzzyb...@gmail.com wrote:

In 0.14.2 the metadata is now being saved on the client non-data node
too. However ClusterHealthRequest for a specific index seems to be not
working as expected now....

I run with an index named "test".
When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

in 0.13.1, if its not connected - check returns RED or index not found
exception - which was great :).

in 0.14.2, if its not connected - it returns GREEN initially
consistently even if the index is not there yet - breaking the check.
Subsequently I tried indexing something. It then created the index
(metadata appears in data directory on the client node), then threw an
exception ( failed to index - expected because there is no data node)

  • and check subsequently returns RED.

On Jan 6, 4:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you try and use 0.14.2, see if it helps? Also, note that you get long
term persistency with the default local gateway as well. So make sure you
want to use the fs gateway.

On Wed, Jan 5, 2011 at 4:44 PM, cwho80 fuzzyb...@gmail.com wrote:

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding data
and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available -
as if they were not created, guessing the metadata isn't read from the
gateway.

I cannot guarantee order of startup if there's a network/power outage.
I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(cwho) #5

When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

I meant, when my client node (ES Client) starts up. sorry for the
typo.

On Jan 7, 10:12 am, cwho80 fuzzyb...@gmail.com wrote:

In 0.14.2 the metadata is now being saved on the client non-data node
too. However ClusterHealthRequest for a specific index seems to be not
working as expected now....

I run with an index named "test".
When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

in 0.13.1, if its not connected - check returns RED or index not found
exception - which was great :).

in 0.14.2, if its not connected - it returns GREEN initially
consistently even if the index is not there yet - breaking the check.
Subsequently I tried indexing something. It then created the index
(metadata appears in data directory on the client node), then threw an
exception ( failed to index - expected because there is no data node)

  • and check subsequently returns RED.

On Jan 6, 4:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you try and use 0.14.2, see if it helps? Also, note that you get long
term persistency with the default local gateway as well. So make sure you
want to use the fs gateway.

On Wed, Jan 5, 2011 at 4:44 PM, cwho80 fuzzyb...@gmail.com wrote:

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding data
and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available -
as if they were not created, guessing the metadata isn't read from the
gateway.

I cannot guarantee order of startup if there's a network/power outage.
I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(Shay Banon) #6

Here is a gist, running with 0.14.2, that creates first a client node, and
then a data node. The client node "data_client" location is never created.
Calling health on the client node before any other node has joined will
cause a MasterNotDiscoveredException (as expected). When adding the data
node, then its data location is created, and you can create an index. At no
point is there data created on the file system for the client node created.
How is it different than your test?

The gist: https://gist.github.com/769484

-shay.banon

On Fri, Jan 7, 2011 at 4:17 AM, cwho80 fuzzybr80@gmail.com wrote:

When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

I meant, when my client node (ES Client) starts up. sorry for the
typo.

On Jan 7, 10:12 am, cwho80 fuzzyb...@gmail.com wrote:

In 0.14.2 the metadata is now being saved on the client non-data node
too. However ClusterHealthRequest for a specific index seems to be not
working as expected now....

I run with an index named "test".
When my data node (ES Client) starts up - I perform a
ClusterHealthRequest("test") regularly (background thread) to verify
if it is connected to a cluster (i.e. ES Server) that can access that
index.

in 0.13.1, if its not connected - check returns RED or index not found
exception - which was great :).

in 0.14.2, if its not connected - it returns GREEN initially
consistently even if the index is not there yet - breaking the check.
Subsequently I tried indexing something. It then created the index
(metadata appears in data directory on the client node), then threw an
exception ( failed to index - expected because there is no data node)

  • and check subsequently returns RED.

On Jan 6, 4:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you try and use 0.14.2, see if it helps? Also, note that you get
long

term persistency with the default local gateway as well. So make sure
you

want to use the fs gateway.

On Wed, Jan 5, 2011 at 4:44 PM, cwho80 fuzzyb...@gmail.com wrote:

I ran a "single-node" ES setup, with the gateway config on my ES
server (eventually it may be replaced by a cluster of boxes with a
shared file system) as:

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store

and it works, I start it up, set up a few indices, (corresponding
data

and metadata appeared in the file location specified) and everything
is available.

Then on another machine - I'll call it the ES Client - I use the Java
API (setting the node on the application to be a non-data client node

  • without any gateway setting).

I noticed that, subsequently messing with the machines:

If the ES Server node starts up first, and ES Client joins its
cluster

  • every thing is fine, indices are available....

If the ES Client node starts up first, and the ES Server node
subsequently starts and join its cluster - indices are not available

as if they were not created, guessing the metadata isn't read from
the

gateway.

I cannot guarantee order of startup if there's a network/power
outage.

I want the index data/metadata to live just on the ES Server, so the
ES Client shouldn't need access to gateway store at /home/escluster
(its just a client and not a data node).

Does anyone has an idea how to get around this or where my
configuration is going wrong? :slight_smile:


(system) #7