Deb package stop service


(Onilton Maciel) #1

Can anyone tell me if in the daemon script of elasticsearch in the deb
package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate
elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to
perform an API shutdown in all nodes (I do a flush before it, but just to
make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when I
bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and
stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid file

exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="cat $PID_FILE"
log_failure_msg "Failed to stop $DESC (pid $PID)"
exit 1
fi
rm -f "$PID_FILE"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e

--


(David Pilato) #2

With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to perform an API shutdown in all nodes (I do a flush before it, but just to make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid file exists, cleaning up"
            elif [ $? -eq 3 ]; then
                    PID="`cat $PID_FILE`"
                    log_failure_msg "Failed to stop $DESC (pid $PID)"
                    exit 1
            fi
            rm -f "$PID_FILE"
    else
            log_progress_msg "(not running)"
    fi
    log_end_msg 0
    set -e

--

--


(Onilton Maciel) #3

I don't remember right now.

But first I usually (when killing programas) send an killall (that a think
that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:

With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb
package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate
elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to
perform an API shutdown in all nodes (I do a flush before it, but just to
make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when
I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and
stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid

file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="cat $PID_FILE"
log_failure_msg "Failed to stop $DESC (pid $PID)"
exit 1
fi
rm -f "$PID_FILE"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e

--

--

--


(Onilton Maciel) #4

Another info, SIGINT seems to cause the same problems of SIGKILL. But a
think SIGINT can be catch by the program, unlike SIGKILL.

So it would be nice if elasticsearch did some handling for the SIGINT
signal.

On Thu, Aug 30, 2012 at 7:00 AM, Onilton Maciel oniltonmaciel@gmail.comwrote:

I don't remember right now.

But first I usually (when killing programas) send an killall (that a think
that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:

With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb
package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate
elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to
perform an API shutdown in all nodes (I do a flush before it, but just to
make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when
I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and
stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid

file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="cat $PID_FILE"
log_failure_msg "Failed to stop $DESC (pid $PID)"
exit 1
fi
rm -f "$PID_FILE"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e

--

--

--


(David Pilato) #5

I never lose a shard when I killed ES or used the service elasticsearch stop command.

I don't understand why you have a missing shard each time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 13:04, Onilton Maciel oniltonmaciel@gmail.com a écrit :

Another info, SIGINT seems to cause the same problems of SIGKILL. But a think SIGINT can be catch by the program, unlike SIGKILL.

So it would be nice if elasticsearch did some handling for the SIGINT signal.

On Thu, Aug 30, 2012 at 7:00 AM, Onilton Maciel oniltonmaciel@gmail.com wrote:
I don't remember right now.

But first I usually (when killing programas) send an killall (that a think that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:
With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to perform an API shutdown in all nodes (I do a flush before it, but just to make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid file exists, cleaning up"
            elif [ $? -eq 3 ]; then
                    PID="`cat $PID_FILE`"
                    log_failure_msg "Failed to stop $DESC (pid $PID)"
                    exit 1
            fi
            rm -f "$PID_FILE"
    else
            log_progress_msg "(not running)"
    fi
    log_end_msg 0
    set -e

--

--

--

--


(Onilton Maciel) #6

I probably used kill -9 (SIGKILL) option when I got the missing shard.

or SIGINT.

Not sure if it's possible to get this error with SIGTERM. (Actually, I
really don't think so.)

Question: Where you still indexing when you killed ES?

On Thu, Aug 30, 2012 at 7:31 AM, David Pilato david@pilato.fr wrote:

I never lose a shard when I killed ES or used the service elasticsearch
stop command.

I don't understand why you have a missing shard each time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 13:04, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Another info, SIGINT seems to cause the same problems of SIGKILL. But a
think SIGINT can be catch by the program, unlike SIGKILL.

So it would be nice if elasticsearch did some handling for the SIGINT
signal.

On Thu, Aug 30, 2012 at 7:00 AM, Onilton Maciel oniltonmaciel@gmail.comwrote:

I don't remember right now.

But first I usually (when killing programas) send an killall (that a
think that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:

With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb
package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate
elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to
perform an API shutdown in all nodes (I do a flush before it, but just to
make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard
when I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and
stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid

file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="cat $PID_FILE"
log_failure_msg "Failed to stop $DESC (pid $PID)"
exit 1
fi
rm -f "$PID_FILE"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e

--

--

--

--

--


(David Pilato) #7

No.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 14:36, Onilton Maciel oniltonmaciel@gmail.com a écrit :

I probably used kill -9 (SIGKILL) option when I got the missing shard.

or SIGINT.

Not sure if it's possible to get this error with SIGTERM. (Actually, I really don't think so.)

Question: Where you still indexing when you killed ES?

On Thu, Aug 30, 2012 at 7:31 AM, David Pilato david@pilato.fr wrote:
I never lose a shard when I killed ES or used the service elasticsearch stop command.

I don't understand why you have a missing shard each time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 13:04, Onilton Maciel oniltonmaciel@gmail.com a écrit :

Another info, SIGINT seems to cause the same problems of SIGKILL. But a think SIGINT can be catch by the program, unlike SIGKILL.

So it would be nice if elasticsearch did some handling for the SIGINT signal.

On Thu, Aug 30, 2012 at 7:00 AM, Onilton Maciel oniltonmaciel@gmail.com wrote:
I don't remember right now.

But first I usually (when killing programas) send an killall (that a think that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:
With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb package)

To everything work as expected when I bring the clusters back, I have to perform an API shutdown in all nodes (I do a flush before it, but just to make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard when I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid file exists, cleaning up"
            elif [ $? -eq 3 ]; then
                    PID="`cat $PID_FILE`"
                    log_failure_msg "Failed to stop $DESC (pid $PID)"
                    exit 1
            fi
            rm -f "$PID_FILE"
    else
            log_progress_msg "(not running)"
    fi
    log_end_msg 0
    set -e

--

--

--

--

--

--


(Onilton Maciel) #8

Question: Where you still indexing when you killed ES?

No.

Ok, so my scenario is a little bit different than yours. =)

So the doubt remains... =/

On Thu, Aug 30, 2012 at 8:44 AM, David Pilato david@pilato.fr wrote:

No.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 14:36, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

I probably used kill -9 (SIGKILL) option when I got the missing shard.

or SIGINT.

Not sure if it's possible to get this error with SIGTERM. (Actually, I
really don't think so.)

Question: Where you still indexing when you killed ES?

On Thu, Aug 30, 2012 at 7:31 AM, David Pilato david@pilato.fr wrote:

I never lose a shard when I killed ES or used the service elasticsearch
stop command.

I don't understand why you have a missing shard each time.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 13:04, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Another info, SIGINT seems to cause the same problems of SIGKILL. But a
think SIGINT can be catch by the program, unlike SIGKILL.

So it would be nice if elasticsearch did some handling for the SIGINT
signal.

On Thu, Aug 30, 2012 at 7:00 AM, Onilton Maciel oniltonmaciel@gmail.comwrote:

I don't remember right now.

But first I usually (when killing programas) send an killall (that a
think that sends a SIGTERM)

And then, if it does not terminate the thing, I usually send a SIGKILL.

BTW, I think this is sort of what what it is done by the init.d script.

On Thu, Aug 30, 2012 at 12:46 AM, David Pilato david@pilato.fr wrote:

With which option do you kill ES node?
-9 ?
-5 ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2012 à 00:51, Onilton Maciel oniltonmaciel@gmail.com a
écrit :

Can anyone tell me if in the daemon script of elasticsearch in the deb
package do the "stop" option works like it's done when I perform a kill?

I worried about this, because almost everytime I use kill to terminate
elasticsearch, I come back with a missing shard. =/

I am talking about a hard install of elastic search (without deb
package)

To everything work as expected when I bring the clusters back, I have
to perform an API shutdown in all nodes (I do a flush before it, but just
to make sure, I don't think it's necessary)

http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-shutdown.html

And then everything works fine when I start the nodes again.

But if kill any (or all) of the nodes, I usually get a missing shard
when I bring the cluster back.

I investigated a little bit in the /etc/init.d/elasticsearch script and
stop doesn't seem to do that:

stop)
log_daemon_msg "Stopping $DESC"

    set +e
    if [ -f "$PID_FILE" ]; then
            start-stop-daemon --stop --pidfile "$PID_FILE" \
                    --user "$ES_USER" \
                    --retry=TERM/20/KILL/5 >/dev/null
            if [ $? -eq 1 ]; then
                    log_progress_msg "$DESC is not running but pid

file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="cat $PID_FILE"
log_failure_msg "Failed to stop $DESC (pid
$PID)"
exit 1
fi
rm -f "$PID_FILE"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e

--

--

--

--

--

--

--


(system) #9