Just Pushed: IP (ipv4) address type

Elastic Stack Elasticsearch
1

Hi,

Just pushed support ip address support (with auto detection):
http://github.com/elasticsearch/elasticsearch/issues/issue/461 (as discussed
in a previous) thread.

Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to", "192.168.0.10").setRefresh(true).execute().actionGet();

    SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))
.execute().actionGet();

-shay.banon

2

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

    "minimum_number_should_match" : 1,
    "boost" : 1.0
}

}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto
detection): Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to", "192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))
.execute().actionGet();

-shay.banon

3

You need to use range query to do range check (can be open ended), not term
query.

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto
detection):
Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))

            .execute().actionGet();

-shay.banon

4

Hi Ted,

may be you should consider better securing of your public service. I was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what kind of
proxy you used, did you use anything except nginx ?)

regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto
detection):
Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))

            .execute().actionGet();

-shay.banon

5

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi Ted,
may be you should consider better securing of your public service. I was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what kind of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection): Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))
.execute().actionGet();

-shay.banon

6

Ted,

can you elaborate more on what exactly you are looking for regarding
security?
I think implementing general security for search server is not easy because
there can be too many different use cases. So I would expect that most of
the security code will be a custom code implemented outside of the search
server.

Regards,
Lukas

On Fri, Oct 29, 2010 at 5:03 PM, Ted Karmel ted.karmel@gmail.com wrote:

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Hi Ted,
may be you should consider better securing of your public service. I was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what kind
of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection):
Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))

            .execute().actionGet();

-shay.banon

7

I'm thinking there will be a Signature header that is sent in the
request to the Elasticsearch server via the REST API.

The signature itself is derived with sha encryption. The encryption
key is a secret access key that resides on both the server initiating
the request and the Elasticsearch server receiving the request. On
the Elasticsearch side, this could reside in the configuration files
in yaml format. The actual data that the key encrypts is the http
request information (e.g. all the headers and the body of the
request). So when the Elasticsearch server receives the request, it
uses the secret access key to parse and decrypt the value of signature
and compares the data that it was sent with the data that is revealed
via decryption. If the same (true), then allow the request to
proceed. If not true, then return an unauthorized http response.

Why do you think this would not be feasible?

On Fri, Oct 29, 2010 at 4:40 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Ted,
can you elaborate more on what exactly you are looking for regarding
security?
I think implementing general security for search server is not easy because
there can be too many different use cases. So I would expect that most of
the security code will be a custom code implemented outside of the search
server.
Regards,
Lukas

On Fri, Oct 29, 2010 at 5:03 PM, Ted Karmel ted.karmel@gmail.com wrote:

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Hi Ted,
may be you should consider better securing of your public service. I was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what kind
of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection): Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))
.execute().actionGet();

-shay.banon

8

Sounds like you want OAUTH, without the nonce protection.

On Fri, Oct 29, 2010 at 11:10 AM, Ted Karmel ted.karmel@gmail.com wrote:

I'm thinking there will be a Signature header that is sent in the
request to the Elasticsearch server via the REST API.

The signature itself is derived with sha encryption. The encryption
key is a secret access key that resides on both the server initiating
the request and the Elasticsearch server receiving the request. On
the Elasticsearch side, this could reside in the configuration files
in yaml format. The actual data that the key encrypts is the http
request information (e.g. all the headers and the body of the
request). So when the Elasticsearch server receives the request, it
uses the secret access key to parse and decrypt the value of signature
and compares the data that it was sent with the data that is revealed
via decryption. If the same (true), then allow the request to
proceed. If not true, then return an unauthorized http response.

Why do you think this would not be feasible?

On Fri, Oct 29, 2010 at 4:40 PM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Ted,
can you elaborate more on what exactly you are looking for regarding
security?
I think implementing general security for search server is not easy
because
there can be too many different use cases. So I would expect that most of
the security code will be a custom code implemented outside of the search
server.
Regards,
Lukas

On Fri, Oct 29, 2010 at 5:03 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Hi Ted,
may be you should consider better securing of your public service. I
was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what
kind
of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for
ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode": "EU"}

While waiting for 0.13, I've given teh following boolean search a
try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection):
Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned
question
(yea.., in java):

    client.prepareIndex("test", "type1", "1").setSource("from",

"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))

            .execute().actionGet();

-shay.banon

9

Well, sort of...

This is what Amazon does for preventing injection attacks on the
buckets of its customers.

Of course, this doesn't address the general hardening of your servers
which is an altogether different subject.

What this prevents is someone figuring out that you use Elasticsearch.
Determing the IP addresses of your front-end servers that send the
requests. Determinining the IP addresses of your Elasticsearch
servers. With that info, they can really mess with your data. The
requests to the ES severs are not dependent on responses. The
responses are only status messages. So you could simulate the IP
address of the front end servers and not be worried about not
receiving a response back.

Naturally, someone could hack into your servers and uncover the secret
access key and do pretty much the same. But, it becomes significantly
more difficult. It is no longer a matter of eavesdropping on HTTP
communication.

I think this option is feasible for Elasticsearch. Already, there is
sha encryption in the Amazon cloud plugins (to backup data to S3). It
would be a matter of parsing the Signature header. If I was any good
at Java, I'd code it myself and try to commit it.

On Fri, Oct 29, 2010 at 5:14 PM, Matt Hartzler matt.hartzler@gmail.com wrote:

Sounds like you want OAUTH, without the nonce protection.

On Fri, Oct 29, 2010 at 11:10 AM, Ted Karmel ted.karmel@gmail.com wrote:

I'm thinking there will be a Signature header that is sent in the
request to the Elasticsearch server via the REST API.

The signature itself is derived with sha encryption. The encryption
key is a secret access key that resides on both the server initiating
the request and the Elasticsearch server receiving the request. On
the Elasticsearch side, this could reside in the configuration files
in yaml format. The actual data that the key encrypts is the http
request information (e.g. all the headers and the body of the
request). So when the Elasticsearch server receives the request, it
uses the secret access key to parse and decrypt the value of signature
and compares the data that it was sent with the data that is revealed
via decryption. If the same (true), then allow the request to
proceed. If not true, then return an unauthorized http response.

Why do you think this would not be feasible?

On Fri, Oct 29, 2010 at 4:40 PM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Ted,
can you elaborate more on what exactly you are looking for regarding
security?
I think implementing general security for search server is not easy
because
there can be too many different use cases. So I would expect that most
of
the security code will be a custom code implemented outside of the
search
server.
Regards,
Lukas

On Fri, Oct 29, 2010 at 5:03 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Hi Ted,
may be you should consider better securing of your public service. I
was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what
kind
of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for
ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode":
"EU"}

While waiting for 0.13, I've given teh following boolean search a
try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection): Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned
question
(yea.., in java):

    client.prepareIndex("test", "type1",

"1").setSource("from",
"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))
.execute().actionGet();

-shay.banon

10

Hi,

Security is a broad area that I do plan to try and tackle at some point
for elasticsearch. There are many areas that this can be applied, but at
least currently, the idea is that this will be solved outside of
elasticsearch.

-shay.banon

On Fri, Oct 29, 2010 at 6:27 PM, Ted Karmel ted.karmel@gmail.com wrote:

Well, sort of...

This is what Amazon does for preventing injection attacks on the
buckets of its customers.

Of course, this doesn't address the general hardening of your servers
which is an altogether different subject.

What this prevents is someone figuring out that you use Elasticsearch.
Determing the IP addresses of your front-end servers that send the
requests. Determinining the IP addresses of your Elasticsearch
servers. With that info, they can really mess with your data. The
requests to the ES severs are not dependent on responses. The
responses are only status messages. So you could simulate the IP
address of the front end servers and not be worried about not
receiving a response back.

Naturally, someone could hack into your servers and uncover the secret
access key and do pretty much the same. But, it becomes significantly
more difficult. It is no longer a matter of eavesdropping on HTTP
communication.

I think this option is feasible for Elasticsearch. Already, there is
sha encryption in the Amazon cloud plugins (to backup data to S3). It
would be a matter of parsing the Signature header. If I was any good
at Java, I'd code it myself and try to commit it.

On Fri, Oct 29, 2010 at 5:14 PM, Matt Hartzler matt.hartzler@gmail.com
wrote:

Sounds like you want OAUTH, without the nonce protection.

On Fri, Oct 29, 2010 at 11:10 AM, Ted Karmel ted.karmel@gmail.com
wrote:

I'm thinking there will be a Signature header that is sent in the
request to the Elasticsearch server via the REST API.

The signature itself is derived with sha encryption. The encryption
key is a secret access key that resides on both the server initiating
the request and the Elasticsearch server receiving the request. On
the Elasticsearch side, this could reside in the configuration files
in yaml format. The actual data that the key encrypts is the http
request information (e.g. all the headers and the body of the
request). So when the Elasticsearch server receives the request, it
uses the secret access key to parse and decrypt the value of signature
and compares the data that it was sent with the data that is revealed
via decryption. If the same (true), then allow the request to
proceed. If not true, then return an unauthorized http response.

Why do you think this would not be feasible?

On Fri, Oct 29, 2010 at 4:40 PM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Ted,
can you elaborate more on what exactly you are looking for regarding
security?
I think implementing general security for search server is not easy
because
there can be too many different use cases. So I would expect that most
of
the security code will be a custom code implemented outside of the
search
server.
Regards,
Lukas

On Fri, Oct 29, 2010 at 5:03 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Hey Lukas,

Yes, it is completely open on purpose for development :slight_smile:

I use nginx 0.8.50 as a front end. Security is an issue I will
definitely tackle. I'm thinking limiting the ip addresses that can
make the requests of course but also adding some key encryption to
authenticate those requests on top of that. Someone could still
ostensibly get a hold of the range of ip addresses and simulate
requests from them for an injection type attack.

I'm thinking something along the lines of what Amazon uses for its
signature (sha1 encryption and base64 encoding of http request data
with a secret key that resides on both ends but not revealed
transparently in communication).

Although, that would be cool if Elasticsearch integrated this sort of
feature... Shay, what do you think?

cheers,
ted

On Fri, Oct 29, 2010 at 8:39 AM, Lukáš Vlček lukas.vlcek@gmail.com
wrote:

Hi Ted,
may be you should consider better securing of your public service.
I
was
able to create a new alias for your twitter index :slight_smile:
(I was just curious how you exposed/secured your REST API and what
kind
of
proxy you used, did you use anything except nginx ?)
regards,
Lukas

On Thu, Oct 28, 2010 at 9:05 PM, Ted Karmel ted.karmel@gmail.com
wrote:

Woohoo! That's great Shay.

Sorry I haven't replied earlier. Yes, AFAIK the data is only for
ipv4
though it will have to deal with ipv6 soon enough.

I was busy on my side setting up the ip address index.

You can give it a spin here :

http://174.129.14.156/geo/ip/

The document schema is :

{"hiend": "0034603007", "countrycode": "FR", "country": "FRANCE",
"lowend": "0033554432", "continent": "EUROPE", "continentcode":
"EU"}

While waiting for 0.13, I've given teh following boolean search a
try.
I suspect I am not expressing the comparison operators correctly.

{
"bool" : {
"must" : {
"term" : { "lowend" =< "3741319167" }
},
{
"must" : {
"term" : { "hiend" >= "3741319167" }
},

   "minimum_number_should_match" : 1,
   "boost" : 1.0

}
}

Any suggestions?

And once again thanks for the push. Awesome!

On Thu, Oct 28, 2010 at 11:05 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Hi,
Just pushed support ip address support (with auto

detection):
Mapper: Ip Type Support (ipv4), auto detection with dynamic mapping · Issue #461 · elastic/elasticsearch · GitHub
(as discussed in a previous) thread.
Here is a simple test I do to have the previous mentioned
question
(yea.., in java):

    client.prepareIndex("test", "type1",

"1").setSource("from",
"192.168.0.5", "to",
"192.168.0.10").setRefresh(true).execute().actionGet();
SearchResponse search = client.prepareSearch()

.setQuery(boolQuery().must(rangeQuery("from").lt("192.168.0.7")).must(rangeQuery("to").gt("192.168.0.7")))

            .execute().actionGet();

-shay.banon