River module stops after nodes shutdown

Elastic Stack Elasticsearch
1

Hi, I have created an elasticsearch river module and I have a question
about its lifecycle.

I met a situation where cluster nodes were shut down before sending
shutdown signal to es-plugin.
Is there any possibility to make my plugin more aware about cluster
state and stop it before nodes will shut down?

If the nodes shut down earlier that my plugin it will throw exceptions
(nodes shutdown) during indexing. In that case nothing bad happens
because data will be indexed next time, but system logs doesn't look
well then :wink:

Thanks for any helpful advice :slight_smile:

2

The close operation on the river will be called when a node shuts down. Are
you saying you don't see it being called?

On Mon, Sep 12, 2011 at 12:21 PM, scoro scoropl@gmail.com wrote:

Hi, I have created an elasticsearch river module and I have a question
about its lifecycle.

I met a situation where cluster nodes were shut down before sending
shutdown signal to es-plugin.
Is there any possibility to make my plugin more aware about cluster
state and stop it before nodes will shut down?

If the nodes shut down earlier that my plugin it will throw exceptions
(nodes shutdown) during indexing. In that case nothing bad happens
because data will be indexed next time, but system logs doesn't look
well then :wink:

Thanks for any helpful advice :slight_smile:

3

Hey Shay,

I see it being called. I just mean I can't index documents in the
interval between shards shutdown and closing river plugin.
I made a mistake and wrote "nodes" instead of "shards" :slight_smile:

On 12 Wrz, 11:23, Shay Banon kim...@gmail.com wrote:

The close operation on the river will be called when a node shuts down. Are
you saying you don't see it being called?

On Mon, Sep 12, 2011 at 12:21 PM, scoro scor...@gmail.com wrote:

Hi, I have created an elasticsearch river module and I have a question
about its lifecycle.

I met a situation where cluster nodes were shut down before sending
shutdown signal to es-plugin.
Is there any possibility to make my plugin more aware about cluster
state and stop it before nodes will shut down?

If the nodes shut down earlier that my plugin it will throw exceptions
(nodes shutdown) during indexing. In that case nothing bad happens
because data will be indexed next time, but system logs doesn't look
well then :wink:

Thanks for any helpful advice :slight_smile:

4

Ahh, yes, you can't index docs when it happens because the rivers will be
closed after other services are closed / stopped. It might make sense to
move this and close the rivers first, allowing one to still operate while
being closed, opened an issue:
Rivers: Close rivers early allowing them to still do index operations · Issue #1330 · elastic/elasticsearch · GitHub.

On Mon, Sep 12, 2011 at 9:17 PM, scoro scoropl@gmail.com wrote:

Hey Shay,

I see it being called. I just mean I can't index documents in the
interval between shards shutdown and closing river plugin.
I made a mistake and wrote "nodes" instead of "shards" :slight_smile:

On 12 Wrz, 11:23, Shay Banon kim...@gmail.com wrote:

The close operation on the river will be called when a node shuts down.
Are
you saying you don't see it being called?

On Mon, Sep 12, 2011 at 12:21 PM, scoro scor...@gmail.com wrote:

Hi, I have created an elasticsearch river module and I have a question
about its lifecycle.

I met a situation where cluster nodes were shut down before sending
shutdown signal to es-plugin.
Is there any possibility to make my plugin more aware about cluster
state and stop it before nodes will shut down?

If the nodes shut down earlier that my plugin it will throw exceptions
(nodes shutdown) during indexing. In that case nothing bad happens
because data will be indexed next time, but system logs doesn't look
well then :wink:

Thanks for any helpful advice :slight_smile:

5

Thank you Shay :wink:

I guess that should help and I will not be concerned about checking
for ability to index my documents after STOP signal :slight_smile:

On 13 Wrz, 01:00, Shay Banon kim...@gmail.com wrote:

Ahh, yes, you can't index docs when it happens because the rivers will be
closed after other services are closed / stopped. It might make sense to
move this and close the rivers first, allowing one to still operate while
being closed, opened an issue:Rivers: Close rivers early allowing them to still do index operations · Issue #1330 · elastic/elasticsearch · GitHub Mon, Sep 12, 2011 at 9:17 PM, scoro scor...@gmail.com wrote:

Hey Shay,

I see it being called. I just mean I can't index documents in the
interval between shards shutdown and closing river plugin.
I made a mistake and wrote "nodes" instead of "shards" :slight_smile:

On 12 Wrz, 11:23, Shay Banon kim...@gmail.com wrote:

The close operation on the river will be called when a node shuts down.
Are
you saying you don't see it being called?

On Mon, Sep 12, 2011 at 12:21 PM, scoro scor...@gmail.com wrote:

Hi, I have created an elasticsearch river module and I have a question
about its lifecycle.

I met a situation where cluster nodes were shut down before sending
shutdown signal to es-plugin.
Is there any possibility to make my plugin more aware about cluster
state and stop it before nodes will shut down?

If the nodes shut down earlier that my plugin it will throw exceptions
(nodes shutdown) during indexing. In that case nothing bad happens
because data will be indexed next time, but system logs doesn't look
well then :wink:

Thanks for any helpful advice :slight_smile: