Any other way to cleanly shut down ElasticSearch other than CTRL+C on a console?

            I'm spawning ElasticSearch through a Process in .NET and I

want to shut it down gracefully (not just kill the process).

            However, the Process class in .NET does not create a process

group which is what is required to use the API to send the console event.

            That said, is there another way to cleanly shut down ES

without using CTRL+C on a console (btw, service installs are out as well,
can't install the service on this machine).

            Thanks in advance.



                            - Nick

Hi Nick

            That said, is there another way to cleanly shut down

ES without using CTRL+C on a console (btw, service installs are out as
well, can’t install the service on this machine).

You can use the shutdown API:

clint

Clinton,

Thank you, that works.

It brings up an interesting question.  How is one supposed to secure access against things like this?  I love ES (even though I can't even begin to get it to run how I want it to in Azure), but security seems to be a big gaping hole that's lacking.

What I'd really like to see is a) basic authentication for the http transport coupled with b) setting a certificate for use for HTTPS

Those two things would go a ^long^ way towards addressing the security issue.

Or am I missing something?

	- Nick

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: Monday, March 14, 2011 11:30 AM
To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down Elasticsearch other than CTRL+C on a console?

Hi Nick

            That said, is there another way to cleanly shut down

ES without using CTRL+C on a console (btw, service installs are out as
well, can’t install the service on this machine).

You can use the shutdown API:

clint

We proxy it. Only allowing safe operations to permiate through haproxy to
the interwebs. The firewall stops direct access. Then, when we want to do
admin things, we just ssh into the boxes.

On Mon, Mar 14, 2011 at 3:49 PM, Administrator admin@sf4answers.com wrote:

Clinton,

   Thank you, that works.

   It brings up an interesting question.  How is one supposed to secure

access against things like this? I love ES (even though I can't even begin
to get it to run how I want it to in Azure), but security seems to be a big
gaping hole that's lacking.

   What I'd really like to see is a) basic authentication for the http

transport coupled with b) setting a certificate for use for HTTPS

   Those two things would go a ^long^ way towards addressing the

security issue.

   Or am I missing something?

           - Nick

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: Monday, March 14, 2011 11:30 AM
To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down Elasticsearch other than
CTRL+C on a console?

Hi Nick

            That said, is there another way to cleanly shut down

ES without using CTRL+C on a console (btw, service installs are out as
well, can’t install the service on this machine).

You can use the shutdown API:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

(we use haproxy as it loadbalances too :smiley: )

On Mon, Mar 14, 2011 at 4:06 PM, Paul Loy keteracel@gmail.com wrote:

We proxy it. Only allowing safe operations to permiate through haproxy to
the interwebs. The firewall stops direct access. Then, when we want to do
admin things, we just ssh into the boxes.

On Mon, Mar 14, 2011 at 3:49 PM, Administrator admin@sf4answers.comwrote:

Clinton,

   Thank you, that works.

   It brings up an interesting question.  How is one supposed to

secure access against things like this? I love ES (even though I can't even
begin to get it to run how I want it to in Azure), but security seems to be
a big gaping hole that's lacking.

   What I'd really like to see is a) basic authentication for the http

transport coupled with b) setting a certificate for use for HTTPS

   Those two things would go a ^long^ way towards addressing the

security issue.

   Or am I missing something?

           - Nick

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: Monday, March 14, 2011 11:30 AM
To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down Elasticsearch other than
CTRL+C on a console?

Hi Nick

            That said, is there another way to cleanly shut down

ES without using CTRL+C on a console (btw, service installs are out as
well, can’t install the service on this machine).

You can use the shutdown API:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Similar here but an SSL connector would be nice so that the transport inside
the datacenter can be secured.

On Mon, Mar 14, 2011 at 4:06 PM, Paul Loy keteracel@gmail.com wrote:

We proxy it. Only allowing safe operations to permiate through haproxy to
the interwebs. The firewall stops direct access. Then, when we want to do
admin things, we just ssh into the boxes.

On Mon, Mar 14, 2011 at 3:49 PM, Administrator admin@sf4answers.comwrote:

Clinton,

   Thank you, that works.

   It brings up an interesting question.  How is one supposed to

secure access against things like this? I love ES (even though I can't even
begin to get it to run how I want it to in Azure), but security seems to be
a big gaping hole that's lacking.

   What I'd really like to see is a) basic authentication for the http

transport coupled with b) setting a certificate for use for HTTPS

   Those two things would go a ^long^ way towards addressing the

security issue.

   Or am I missing something?

           - Nick

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: Monday, March 14, 2011 11:30 AM
To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down Elasticsearch other than
CTRL+C on a console?

Hi Nick

            That said, is there another way to cleanly shut down

ES without using CTRL+C on a console (btw, service installs are out as
well, can’t install the service on this machine).

You can use the shutdown API:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

On Mon, 2011-03-14 at 16:06 +0000, Paul Loy wrote:

We proxy it. Only allowing safe operations to permiate through haproxy
to the interwebs. The firewall stops direct access. Then, when we want
to do admin things, we just ssh into the boxes.

On Mon, Mar 14, 2011 at 3:49 PM, Administrator admin@sf4answers.com
wrote:
Clinton,

           Thank you, that works.
    
           It brings up an interesting question.  How is one
    supposed to secure access against things like this?  I love ES
    (even though I can't even begin to get it to run how I want it
    to in Azure), but security seems to be a big gaping hole
    that's lacking.
    
           What I'd really like to see is a) basic authentication
    for the http transport coupled with b) setting a certificate
    for use for HTTPS
    
           Those two things would go a ^long^ way towards
    addressing the security issue.
    
           Or am I missing something?

Hello,

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

--
Daniel Maher
« can't talk, too busy calculating computrons. »

Hi Daniel

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

I both agree and disagree :slight_smile:

I agree: the ES code shouldn't be bogged down by authentication as a
standard.

However, this is a frequently requested piece of functionality,
(especially given that ES is supposed to run in the cloud).

While proxying it is easy, the user still has to figure out which proxy
they want to use, how to configure it, then they have two run two
separate services.

I think making authz/authen available as a plugin would be very useful.

clint

Heya,

Transport level encryption is actually not that difficult to add (SSL). The more complex feature is authentication / authorization. Once we implement that, then we need to have role base support (admin can call shutdown, while other user roles can not), and of course, where is all this info stored (and probably, integration with external user management systems).

I don't think proxiying is that difficult to do, and even if elasticsearch supports SSL, I personally will probably proxy it with another system that does the SSL. The fact that it can run in the "cloud" does not mean that it needs to have those features.

-shay.banon
On Monday, March 14, 2011 at 6:42 PM, Clinton Gormley wrote:
Hi Daniel

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

I both agree and disagree :slight_smile:

I agree: the ES code shouldn't be bogged down by authentication as a
standard.

However, this is a frequently requested piece of functionality,
(especially given that ES is supposed to run in the cloud).

While proxying it is easy, the user still has to figure out which proxy
they want to use, how to configure it, then they have two run two
separate services.

I think making authz/authen available as a plugin would be very useful.

clint

On Mar 14, 12:29 pm, Daniel Maher dma...@milestonelab.com wrote:

Hello,

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

I am also in agreement that ES does not need its own authentication as
a first-class requirement. Perhaps as a plugin.

Many alternative datastores/frameworks, such as Hadoop and MongoDB, do
not have authentication built-in. Having security provided at the
machine/network level is not only an acceptable use-case, but perhaps
might become the dominant one.

Ivan

Shay,

            Respectfully, I think that you understate the last point about running in the cloud; yes, if you are proxying ES then SSL isn’t that much of an important feature, you just have to harden it from access from other computers in the same subnet.



            However, if you wanted to have a direct-facing instance of ES hosted in the cloud, then SSL would most definitely be a requirement.,



            In regards to authentication, there is a bit of confusion here.



            The HTTP 1.1 spec (see: http://www.ietf.org/rfc/rfc2616.txt section 14.8 <http://www.ietf.org/rfc/rfc2616.txt%20section%2014.8> ) is titled “Authorization”.  However, this is a bit of a misnomer, IMO.  It should really be titled “Authentication” as it relates to credentials and how the user identifies itself to the server according to the HTTP protocol.  Authorization is dependent on authentication (you have to know whom you are dealing with), but authentication is not based on authorization.



            That said, when I refer to “Basic Authentication”, I’m really referring to the Basic Authentication Scheme in HTTP (http://www.webdav.org/specs/rfc2617.html#rfc.section.2), not custom authorization schemes based on authentication.



            Now if both SSL and Basic Authentication were provided (as per the HTTP specification), it would harden up a direct-facing instance of ES quite nicely; you could use a username/password combination, as well as a certificate to encrypt everything that is passed between the endpoints.  Yes, the weak point is the username/password combo, but at least it wouldn’t be able to be viewed in plain text going across the wire, and you could block out anyone who doesn’t have it.



            Note, I’m not asking for anything other than what the HTTP protocol specifies in these two very specific areas; custom authorization (and authentication as well) is a PITA, many people have many different views on how this should be done and there will be no one-size-fits-all solution.



                            - Nick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Monday, March 14, 2011 1:46 PM
To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down ElasticSearch other than CTRL+C on a console?

Heya,

Transport level encryption is actually not that difficult to add (SSL). The more complex feature is authentication / authorization. Once we implement that, then we need to have role base support (admin can call shutdown, while other user roles can not), and of course, where is all this info stored (and probably, integration with external user management systems).

I don't think proxiying is that difficult to do, and even if elasticsearch supports SSL, I personally will probably proxy it with another system that does the SSL. The fact that it can run in the "cloud" does not mean that it needs to have those features.

-shay.banon

On Monday, March 14, 2011 at 6:42 PM, Clinton Gormley wrote:

Hi Daniel

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

I both agree and disagree :slight_smile:

I agree: the ES code shouldn't be bogged down by authentication as a
standard.

However, this is a frequently requested piece of functionality,
(especially given that ES is supposed to run in the cloud).

While proxying it is easy, the user still has to figure out which proxy
they want to use, how to configure it, then they have two run two
separate services.

I think making authz/authen available as a plugin would be very useful.

clint

Nick,

I think the issue is that simple HTTP authentication does not solve many of
the use cases and immediately other requirements come into play. For
example, there are destructive API calls, you want to authenticate users but
likely would never want to allow users to do these type of operations hence
the need to have users, groups, roles, etc.

In the use cases we've seen so far, you always need to front ES with another
system to provide these capabilities. It is definitely an option to add them
to ES, but them you have bloat, etc. and as you mention, hard to find
consensus and provide full functionality.

Regardless of whether you run ES in the cloud or inside the firewall,
currently best practice is to have another system to provide non-search
related functions, including authentication and authorization.
Implementing it as a plugin would be a good approach provided that it is
technically viable. I think what other people are saying is that we rather
have Shay work on core features than build the peripheral functionality.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Mon, Mar 14, 2011 at 2:17 PM, Administrator admin@sf4answers.com wrote:

Shay,

            Respectfully, I think that you understate the last point

about running in the cloud; yes, if you are proxying ES then SSL isn’t that
much of an important feature, you just have to harden it from access from
other computers in the same subnet.

            However, if you wanted to have a direct-facing instance of

ES hosted in the cloud, then SSL would most definitely be a
requirement.,

            In regards to authentication, there is a bit of confusion

here.

            The HTTP 1.1 spec (see: http://www.ietf.org/rfc/rfc2616.txt

section 14.8) is titled “Authorization”. However, this is a bit of a
misnomer, IMO. It should really be titled “Authentication” as it relates to
credentials and how the user identifies itself to the server according to
the HTTP protocol. Authorization is dependent on authentication (you have
to know whom you are dealing with), but authentication is not based on
authorization.

            That said, when I refer to “Basic Authentication”, I’m

really referring to the Basic Authentication Scheme in HTTP (
HTTP Authentication: Basic and Digest Access Authentication), not custom authorization
schemes based on authentication.

            Now if both SSL and Basic Authentication were provided (as

per the HTTP specification), it would harden up a direct-facing instance of
ES quite nicely; you could use a username/password combination, as well as a
certificate to encrypt everything that is passed between the endpoints.
Yes, the weak point is the username/password combo, but at least it wouldn’t
be able to be viewed in plain text going across the wire, and you could
block out anyone who doesn’t have it.

            Note, I’m *not* asking for anything other than what the

HTTP protocol specifies in these two very specific areas; custom
authorization (and authentication as well) is a PITA, many people have many
different views on how this should be done and there will be no
one-size-fits-all solution.

                            - Nick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Monday, March 14, 2011 1:46 PM

To: users@elasticsearch.com
Subject: Re: Any other way to cleanly shut down Elasticsearch other than
CTRL+C on a console?

Heya,

Transport level encryption is actually not that difficult to add (SSL).
The more complex feature is authentication / authorization. Once we
implement that, then we need to have role base support (admin can call
shutdown, while other user roles can not), and of course, where is all this
info stored (and probably, integration with external user management
systems).

I don't think proxiying is that difficult to do, and even if
elasticsearch supports SSL, I personally will probably proxy it with another
system that does the SSL. The fact that it can run in the "cloud" does not
mean that it needs to have those features.

-shay.banon

On Monday, March 14, 2011 at 6:42 PM, Clinton Gormley wrote:

Hi Daniel

I'm not sure that ES needs to have any sort of authentication or SSL
features; as Paul Loy pointed out, it's trivial to proxy ES through a
mechanism that is already designed to do those sorts of things, so what
would be the interest in bogging down the ES code base with superfluous
functionality ?

I both agree and disagree :slight_smile:

I agree: the ES code shouldn't be bogged down by authentication as a
standard.

However, this is a frequently requested piece of functionality,
(especially given that ES is supposed to run in the cloud).

While proxying it is easy, the user still has to figure out which proxy
they want to use, how to configure it, then they have two run two
separate services.

I think making authz/authen available as a plugin would be very useful.

clint

Forgot to add a link regarding (lack of) authentication:
http://pl.atyp.us/wordpress/?p=2988

I am not a Windows users, but on *nix, I use kill -2 to kill an non-
service Elasticsearch process.

On Mar 14, 2:01 pm, Ivan Brusic ivan_bru...@yahoo.com wrote:

I am also in agreement that ES does not need its own authentication as
a first-class requirement. Perhaps as a plugin.

Many alternative datastores/frameworks, such as Hadoop and MongoDB, do
not have authentication built-in. Having security provided at the
machine/network level is not only an acceptable use-case, but perhaps
might become the dominant one.

Ivan