Logstash Geohash Question

David_Kleiner · March 6, 2015, 10:10pm

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · March 7, 2015, 12:26am

Can you elaborate? I use the geoip filter all the time with kb4 and have no
problems
On 07/03/2015 9:10 am, "David Kleiner" david.kleiner@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_ZMU0XpJ-4ZiCacKjjqOaXxF4PE9FMTbDQdk0-GdXgsg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

David_Kleiner · March 7, 2015, 12:38am

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" <david....@gmail.com <javascript:>>
wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

David_Kleiner · March 7, 2015, 12:50am

And here are my geoip fields:

geoip: {

properties: {
- dma_code: {
  - type: long
    }
- continent_code: {
  - type: string
    }
- city_name: {
  - type: string
    }
- location: {
  - type: double
    }
- area_code: {
  - type: long
    }
- country_name: {
  - type: string
    }
- ip: {
  - type: string
    }
- region_name: {
  - type: string
    }
- timezone: {
  - type: string
    }
- postal_code: {
  - type: string
    }
- country_code2: {
  - type: string
    }
- country_code3: {
  - type: string
    }
- longitude: {
  - type: double
    }
- latitude: {
  - type: double
    }
- real_region_name: {
  - type: string
    }
    }

}

On Friday, March 6, 2015 at 4:38:56 PM UTC-8, David Kleiner wrote:

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" david....@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1e0688ed-d97e-474a-a44f-68cfbfd04922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · March 7, 2015, 3:39pm

ES needs a single lat+lon field to read. It or KB won't combine things.
On 06/03/2015 4:38 pm, "David Kleiner" david.kleiner@gmail.com wrote:

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" david....@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-8UUh9NB1SWFOUgxPg3K74mvFRpsxA9ucLGG%3DmOEAkmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Michael_6 · March 17, 2015, 4:07pm

What do you mean exactly?

These are the fields I'm able to obtain, whereas geoip.coordinates is built
by using

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

in my logstash.conf.

geoip.city_name Warsawtgeoip.continent_code EU*#geoip.coordinates
["21.0","52.25"]tgeoip.country_code2 PLtgeoip.country_code3 POLt*
geoip.country_name Polandtgeoip.ip 217.67.205.50*#geoip.latitude 52.25#*
geoip.location [21,52.25]#geoip.longitude 21tgeoip.real_region_name
Mazowieckietgeoip.region_name 78tgeoip.timezone Europe/Warsaw
Can please be so kind and post the part of your geoip filter in your
logstash.conf where handle the building of fields in order to use in tile
map of kb4?

Thanks in advance

Am Samstag, 7. März 2015 16:40:07 UTC+1 schrieb Mark Walkom:

ES needs a single lat+lon field to read. It or KB won't combine things

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · March 17, 2015, 4:44pm

It'll be able to read geoip.coordinates if you point to it.

On 17 March 2015 at 09:07, Michael bunszz@gmail.com wrote:

What do you mean exactly?

These are the fields I'm able to obtain, whereas geoip.coordinates is
built by using

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

in my logstash.conf.

geoip.city_name Warsawtgeoip.continent_code EU*#geoip.coordinates
["21.0","52.25"]tgeoip.country_code2 PLtgeoip.country_code3 POLt*
geoip.country_name Polandtgeoip.ip 217.67.205.50*#geoip.latitude 52.25
#geoip.location [21,52.25]#geoip.longitude 21tgeoip.real_region_name
Mazowieckietgeoip.region_name 78t*geoip.timezone Europe/Warsaw
Can please be so kind and post the part of your geoip filter in your
logstash.conf where handle the building of fields in order to use in tile
map of kb4?

Thanks in advance

Am Samstag, 7. März 2015 16:40:07 UTC+1 schrieb Mark Walkom:

ES needs a single lat+lon field to read. It or KB won't combine things

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9ytHKCe5SkTwC9ZPkSBkA%2Bn8cTdBCZq99TZqYqA7cCCg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Michael_6 · March 18, 2015, 9:45am

Unfortunately not. I have the same problem what David described with his
screenshot. The only aggregation that shows up in the left panel for geo
coordinates is of type geohash and below there is no field to choose at all.

Am Dienstag, 17. März 2015 17:44:32 UTC+1 schrieb Mark Walkom:

It'll be able to read geoip.coordinates if you point to it.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5053a865-9f16-414d-a253-0ebe30de885c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Michael_6 · March 18, 2015, 10:02am

Do I have to add some extra fields to the coordinates field as described
in http://www.tagwith.com/question_345822_kibana-3-geojson-vs-kibana4-geohash/
?
Problem here is that the logstash.conf does not seem to like 3-dim arrays
...
add_field => [ "[geoip][coordinates][lat_lon]", true ] does not work at
all. Perhaps I'mm wrong with the syntax?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18e0d443-c143-46b7-839c-612d4b0028fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Michael_6 · March 18, 2015, 2:36pm

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create a
dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described here
http://logstash.net/docs/1.4.2/filters/geoip) is not taken into account.
So, it may be possible to manually create a mapping template (see
http://logstash.net/docs/1.4.2/filters/geoip ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · March 18, 2015, 8:46pm

That sounds really weird, I have a non-LS index with geopoints and I can
definitely read and plot these in KB4.

Are you running the latest KB4 release? What version of ES?

On 18 March 2015 at 07:36, Michael bunszz@gmail.com wrote:

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create a
dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described here
Geoip filter plugin | Logstash Reference [8.11] | Elastic) is not taken into account.
So, it may be possible to manually create a mapping template (see
Geoip filter plugin | Logstash Reference [8.11] | Elastic ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9-wZdUYCG3pBOFgauBeHSKhy1NONh_n7nPJLZ%3Dqf-W2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

David_Kleiner · March 18, 2015, 9:01pm

In my case, it's ES 1.4.4, KB4 4.0.1, I use logstash-forwarder to
logstash-forwarder input, rabbitmq output, geoip filter in logstash and ES
river plugin, no template.

Perhaps I should've created a dedicated template but I now have 3 years
worth of index data and reindexing it will be painful.

On Wednesday, March 18, 2015 at 1:46:51 PM UTC-7, Mark Walkom wrote:

That sounds really weird, I have a non-LS index with geopoints and I can
definitely read and plot these in KB4.

Are you running the latest KB4 release? What version of ES?

On 18 March 2015 at 07:36, Michael <bun...@gmail.com <javascript:>> wrote:

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create
a dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described
here Geoip filter plugin | Logstash Reference [8.11] | Elastic) is not taken into
account.
So, it may be possible to manually create a mapping template (see
Geoip filter plugin | Logstash Reference [8.11] | Elastic ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4c92e0d1-68f4-4e45-af96-d89eac4bd914%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Geohash Filter in kibana and Custom layers in maps Elasticsearch	1	374	July 6, 2017
Kibana geo polygon support Elasticsearch	1	438	July 6, 2017
[Solved] Canno't see geoip.location in visualization Kibana	5	383	October 16, 2019
GEOIP Enable Logstash	25	3985	March 9, 2017
GeoIP lookup without logstash or ingestnode pipeline Kibana	6	403	May 9, 2018

Logstash Geohash Question

Related Topics