IP geolocation without Logstash


(Justin Koehler) #1

I'm working on a system to record usage data for an application that
submits its data to an ES cluster. I would like to record the location of
each data point based on IP geolocation. I found the Logstash plugin that
uses the GeoIP databases, but I was unable to find any solutions built for
just Elasticsearch. Has anybody done something like this before?

In addition, it would be convenient to extract the IP of the point itself
from the "X-Forwarded-For" header of the incoming data point. Is there a
way to access these headers when the point is received by Elasticsearch?

Thanks in advance for any help.

Justin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7cb0010-103c-4ff7-8cd7-f5da5188f9bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Otis Gospodnetić) #2

Hi,

On Thursday, July 17, 2014 10:20:55 AM UTC-4, Justin Koehler wrote:

I'm working on a system to record usage data for an application that
submits its data to an ES cluster. I would like to record the location of
each data point based on IP geolocation. I found the Logstash plugin that
uses the GeoIP databases, but I was unable to find any solutions built for
just Elasticsearch. Has anybody done something like this before?

This is something that's typically done outside ES, in a document
processing pipeline or indexer.

In addition, it would be convenient to extract the IP of the point itself

from the "X-Forwarded-For" header of the incoming data point. Is there a
way to access these headers when the point is received by Elasticsearch?

Doable with a custom Rest Action.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa71ee2b-9894-4568-95f7-3be0e5b0738c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alexander Reelsen) #3

Hey,

this is exactly what logstash is for, so you may want to give it a try, as
it is already there. :slight_smile:
Also you can use the geoip filter to extract the ip address from the header
as well, granted you log that one.

--Alex

On Sat, Jul 19, 2014 at 6:26 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

On Thursday, July 17, 2014 10:20:55 AM UTC-4, Justin Koehler wrote:

I'm working on a system to record usage data for an application that
submits its data to an ES cluster. I would like to record the location of
each data point based on IP geolocation. I found the Logstash plugin that
uses the GeoIP databases, but I was unable to find any solutions built for
just Elasticsearch. Has anybody done something like this before?

This is something that's typically done outside ES, in a document
processing pipeline or indexer.

In addition, it would be convenient to extract the IP of the point itself

from the "X-Forwarded-For" header of the incoming data point. Is there a
way to access these headers when the point is received by Elasticsearch?

Doable with a custom Rest Action.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa71ee2b-9894-4568-95f7-3be0e5b0738c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/aa71ee2b-9894-4568-95f7-3be0e5b0738c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_pMUstvq7dLJPhwY7iK5-TKm59tHsRk4ZHYutcUvzE0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Andreas Lehr) #4

Hi Alex,

how exactly could this work?
For example we are using the pattern "Quotedstring" to extract the up to 4
IPs in the X-Forwared-For header of our Apache Logs.
When we then try using this one in the geoip filter the filter seems to
miss the IP.

example:

grok {
type => http_log
patterns_dir => [ "/opt/logstash/patterns" ]
pattern => "%{APACHELOG}"
}
geoip {
source => "xforwardedfor_header"
fields => [ "city_name", "country_code2", "country_name", "location",
"real_region_name", "postal_code" ]
add_tag => [ "geoip" ]
}
....
patter excerpt:
APACHELOG %{QUOTEDSTRING:xforwardedfor_header} [%{HTTPDATE:time}]

Thanks!

Am Montag, 4. August 2014 09:56:53 UTC+2 schrieb Alexander Reelsen:

Hey,

this is exactly what logstash is for, so you may want to give it a try, as
it is already there. :slight_smile:
Also you can use the geoip filter to extract the ip address from the
header as well, granted you log that one.

--Alex

On Sat, Jul 19, 2014 at 6:26 AM, Otis Gospodnetic <otis.gos...@gmail.com
<javascript:>> wrote:

Hi,

On Thursday, July 17, 2014 10:20:55 AM UTC-4, Justin Koehler wrote:

I'm working on a system to record usage data for an application that
submits its data to an ES cluster. I would like to record the location of
each data point based on IP geolocation. I found the Logstash plugin that
uses the GeoIP databases, but I was unable to find any solutions built for
just Elasticsearch. Has anybody done something like this before?

This is something that's typically done outside ES, in a document
processing pipeline or indexer.

In addition, it would be convenient to extract the IP of the point itself

from the "X-Forwarded-For" header of the incoming data point. Is there a
way to access these headers when the point is received by Elasticsearch?

Doable with a custom Rest Action.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa71ee2b-9894-4568-95f7-3be0e5b0738c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/aa71ee2b-9894-4568-95f7-3be0e5b0738c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0d901d63-ee7b-4586-a8b6-60313ab16c85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5