Logstash Geohash Question

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you elaborate? I use the geoip filter all the time with kb4 and have no
problems
On 07/03/2015 9:10 am, "David Kleiner" david.kleiner@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_ZMU0XpJ-4ZiCacKjjqOaXxF4PE9FMTbDQdk0-GdXgsg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" <david....@gmail.com <javascript:>>
wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

And here are my geoip fields:

geoip: {

  • properties: {
    • dma_code: {
      • type: long
        }
    • continent_code: {
      • type: string
        }
    • city_name: {
      • type: string
        }
    • location: {
      • type: double
        }
    • area_code: {
      • type: long
        }
    • country_name: {
      • type: string
        }
    • ip: {
      • type: string
        }
    • region_name: {
      • type: string
        }
    • timezone: {
      • type: string
        }
    • postal_code: {
      • type: string
        }
    • country_code2: {
      • type: string
        }
    • country_code3: {
      • type: string
        }
    • longitude: {
      • type: double
        }
    • latitude: {
      • type: double
        }
    • real_region_name: {
      • type: string
        }
        }

}

On Friday, March 6, 2015 at 4:38:56 PM UTC-8, David Kleiner wrote:

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" david....@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1e0688ed-d97e-474a-a44f-68cfbfd04922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES needs a single lat+lon field to read. It or KB won't combine things.
On 06/03/2015 4:38 pm, "David Kleiner" david.kleiner@gmail.com wrote:

Sure.

When I try to create a tile map, the only aggregation field that shows up
in the left panel for geo coordinates is of type geohash, which I don't
have in the index, see screenshot.

On Friday, March 6, 2015 at 4:27:01 PM UTC-8, Mark Walkom wrote:

Can you elaborate? I use the geoip filter all the time with kb4 and have
no problems
On 07/03/2015 9:10 am, "David Kleiner" david....@gmail.com wrote:

Hi,

Are there any plans to add calculating geohash to Logstash? Now that
Kibana4 is out and won't accept geo coordinates for its map
visualization...

Thanks,

David

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/47a6dfbe-b7bd-4de0-8720-ab34cc5eb350%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5ed8c78b-761b-4b14-b18e-f9b16d85d828%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-8UUh9NB1SWFOUgxPg3K74mvFRpsxA9ucLGG%3DmOEAkmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

What do you mean exactly?

These are the fields I'm able to obtain, whereas geoip.coordinates is built
by using

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

in my logstash.conf.

geoip.city_name Warsawtgeoip.continent_code EU*#geoip.coordinates
["21.0","52.25"]tgeoip.country_code2 PL
tgeoip.country_code3 POLt*
geoip.country_name Polandtgeoip.ip 217.67.205.50*#geoip.latitude 52.25#*
geoip.location [21,52.25]#geoip.longitude 21tgeoip.real_region_name
Mazowieckietgeoip.region_name 78tgeoip.timezone Europe/Warsaw
Can please be so kind and post the part of your geoip filter in your
logstash.conf where handle the building of fields in order to use in tile
map of kb4?

Thanks in advance

Am Samstag, 7. März 2015 16:40:07 UTC+1 schrieb Mark Walkom:

ES needs a single lat+lon field to read. It or KB won't combine things

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It'll be able to read geoip.coordinates if you point to it.

On 17 March 2015 at 09:07, Michael bunszz@gmail.com wrote:

What do you mean exactly?

These are the fields I'm able to obtain, whereas geoip.coordinates is
built by using

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

in my logstash.conf.

geoip.city_name Warsawtgeoip.continent_code EU*#geoip.coordinates
["21.0","52.25"]tgeoip.country_code2 PL
tgeoip.country_code3 POLt*
geoip.country_name Polandtgeoip.ip 217.67.205.50*#geoip.latitude 52.25
#geoip.location [21,52.25]#geoip.longitude 21tgeoip.real_region_name
Mazowieckie
tgeoip.region_name 78t*geoip.timezone Europe/Warsaw
Can please be so kind and post the part of your geoip filter in your
logstash.conf where handle the building of fields in order to use in tile
map of kb4?

Thanks in advance

Am Samstag, 7. März 2015 16:40:07 UTC+1 schrieb Mark Walkom:

ES needs a single lat+lon field to read. It or KB won't combine things

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/633b740d-1f9d-43ed-acdd-670e30829503%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9ytHKCe5SkTwC9ZPkSBkA%2Bn8cTdBCZq99TZqYqA7cCCg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Unfortunately not. I have the same problem what David described with his
screenshot. The only aggregation that shows up in the left panel for geo
coordinates is of type geohash and below there is no field to choose at all.

Am Dienstag, 17. März 2015 17:44:32 UTC+1 schrieb Mark Walkom:

It'll be able to read geoip.coordinates if you point to it.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5053a865-9f16-414d-a253-0ebe30de885c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do I have to add some extra fields to the coordinates field as described
in http://www.tagwith.com/question_345822_kibana-3-geojson-vs-kibana4-geohash/
?
Problem here is that the logstash.conf does not seem to like 3-dim arrays
...
add_field => [ "[geoip][coordinates][lat_lon]", true ] does not work at
all. Perhaps I'mm wrong with the syntax?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18e0d443-c143-46b7-839c-612d4b0028fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create a
dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described here
http://logstash.net/docs/1.4.2/filters/geoip) is not taken into account.
So, it may be possible to manually create a mapping template (see
http://logstash.net/docs/1.4.2/filters/geoip ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

That sounds really weird, I have a non-LS index with geopoints and I can
definitely read and plot these in KB4.

Are you running the latest KB4 release? What version of ES?

On 18 March 2015 at 07:36, Michael bunszz@gmail.com wrote:

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create a
dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described here
Geoip filter plugin | Logstash Reference [8.11] | Elastic) is not taken into account.
So, it may be possible to manually create a mapping template (see
Geoip filter plugin | Logstash Reference [8.11] | Elastic ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9-wZdUYCG3pBOFgauBeHSKhy1NONh_n7nPJLZ%3Dqf-W2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

In my case, it's ES 1.4.4, KB4 4.0.1, I use logstash-forwarder to
logstash-forwarder input, rabbitmq output, geoip filter in logstash and ES
river plugin, no template.

Perhaps I should've created a dedicated template but I now have 3 years
worth of index data and reindexing it will be painful.

On Wednesday, March 18, 2015 at 1:46:51 PM UTC-7, Mark Walkom wrote:

That sounds really weird, I have a non-LS index with geopoints and I can
definitely read and plot these in KB4.

Are you running the latest KB4 release? What version of ES?

On 18 March 2015 at 07:36, Michael <bun...@gmail.com <javascript:>> wrote:

Ok, seems I resolved the issue:

In short: You have to use the default output index in logstash:

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

in ES this results in the index logstash-YYYY.MM.DD

Since I try to deal with syslogs, I thought it was a good idea to create
a dedicated index.

output {
if [type] == "syslog" {
elasticsearch {
host => "localhost"
index => "syslog-%{+YYYY.MM.dd}"
protocol => "http"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

The thing is: Now the autmatic mapping feature of logstash (described
here Geoip filter plugin | Logstash Reference [8.11] | Elastic) is not taken into
account.
So, it may be possible to manually create a mapping template (see
Geoip filter plugin | Logstash Reference [8.11] | Elastic ... manage_template /
template) in the logstash output config.
Or you can stick to the default output index of logstash.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d0cda794-a6cd-4f45-bf46-38cf80967c74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4c92e0d1-68f4-4e45-af96-d89eac4bd914%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.