Illegal latitude value exception in elasticsearch

Elastic Stack Elasticsearch
21

I think I have a problem with the parsing. Can it be the reason?

22

I meant:

  • a JSon document as it is when it reaches the output stage of logstash (use stdout plugin with a json codec)
  • Elasticsearch mapping
23

Hi,

After hours of trying to understand what is wrong, I am now on the right direction I think.

Anyway,

I added the following to my mapping: "clientGeoLocation":{ "type": "geo_point"}

and in the logstash filter I did that:

        geoip {
               source => "clientrealip"
               add_field => ["clientGeoLocation" , "%{[geoip][location]}"]
        }

In the Kibana discover results I see the following:

clientGeoLocation

But when I am trying to visualize it, it looks like that:

geoiplocation-onmap

Any idea?

Thanks
Sharon.

24

Can you share the full output of a JSON document?

25

Here are one example:

  1. The Exception:
    Caused by: java.lang.IllegalArgumentException: illegal latitude value [-93.902] for clientGeoLocation

    The Json output from logstash:

{"loadbalancerip":"10.108.2.44","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access4.log","type":"omniwebaccss","bn":"st22","wod":"1","issuer":"r","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-93.902,32.403","apMacId":"10:56:11:15:d3:8d","timestamp":"26/Jul/2017:00:00:00","a":"as","geoip":{"timezone":"America/Chicago","ip":"73.91.231.150","latitude":32.403,"continent_code":"NA","city_name":"Shreveport","country_code2":"US","country_name":"United States","dma_code":612,"country_code3":"US","region_name":"Louisiana","location":[-93.902,32.403],"postal_code":"71129","longitude":-93.902,"region_code":"LA"},"offset":258,"macId":"90:c7:d8:fa:78:e9","input_type":"log","message":"10.108.2.44 "73.91.231.150, 10.10.10.165" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/landing?c=e&macId=90%3Ac7%3Ad8%3Afa%3A78%3Ae9&location=WODTrial&apMacId=10%3A56%3A11%3A15%3Ad3%3A8d&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Jasper+LTE HTTP/1.1" 303 -\n\n","kvstring":"macId=90%3Ac7%3Ad8%3Afa%3A78%3Ae9&location=WODTrial&apMacId=10%3A56%3A11%3A15%3Ad3%3A8d&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Jasper+LTE","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.165","httpstatuscode":"303","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.91.231.150","httpversion":"1.1","location":"WODTrial","deviceModel":"ZTE+Jasper+LTE","page":"/wod/landing?c=e"}

In the exception it is writtenillegal latitude value but in fact the -93.902 is the longitude in the geoip in the output json from logstash.

What am I missing here? Maybe it is connected to the field clientGeoLocation?

From my logstash.conf file:

            geoip {
                   source => "clientrealip"
                   add_field => ["clientGeoLocation" , "%{[geoip][location]}"]
            }

Thanks
Sharon.

26

Just more details to make my problem clear:

I have 7 events.

3 of them were rejected with the latitude exception, as shown in previous replay.

4 of them were loaded to db with no exception, but they are out of boundaries of the map, as shown in the replay with the map draw.

The full Json stuck is: (In two messages because of 7000 characters limitation)

{"loadbalancerip":"10.108.2.44","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access8.log","type":"omniwebaccss","bn":"st22","w
od":"1","issuer":"r","httppayload":"1489","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpela
stic","clientGeoLocation":"-81.3029,29.7596","apMacId":"fc:51:a4:57:f5:12","timestamp":"26/Jul/2017:00:00:00","a":"as","geoip":{"timezone":
"America/New_York","ip":"73.35.67.125","latitude":29.7596,"continent_code":"NA","city_name":"St. Augustine","country_code2":"US","country_n
ame":"United States","dma_code":561,"country_code3":"US","region_name":"Florida","location":[-81.3029,29.7596],"postal_code":"32086","longi
tude":-81.3029,"region_code":"FL"},"offset":269,"macId":"8c:79:67:6c:c6:d8","input_type":"log","message":"10.108.2.44 \"73.35.67.125, 10.10
.10.157\" - - [26/Jul/2017:00:00:00 -0500] \"GET /wod/static/welcome.html?c=e&macId=8c%3A79%3A67%3A6c%3Ac6%3Ad8&location=default&apMacId=fc
%3A51%3Aa4%3A57%3Af5%3A12&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Obsidian HTTP/1.1\" 200 1489\n\n","kvstring":"macId=8c%3A79%3A67%3A6c
%3Ac6%3Ad8&location=default&apMacId=fc%3A51%3Aa4%3A57%3Af5%3A12&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Obsidian","tags":["beats_input_
codec_plain_applied"],"clientnattedip":"10.10.10.157","httpstatuscode":"200","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.35.
67.125","httpversion":"1.1","location":"default","deviceModel":"ZTE+Obsidian","page":"/wod/static/welcome.html?c=e"}{"loadbalancerip":"10.1
08.2.44","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access1.log","type":"omniwebaccss","bn":"st22","wod":"1","issuer":"r","@
version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-97.822,37.751","
apMacId":"c4:27:95:56:d2:e3","timestamp":"26/Jul/2017:00:00:00","a":"as","geoip":{"ip":"73.112.77.205","latitude":37.751,"country_code2":"U
S","country_name":"United States","continent_code":"NA","country_code3":"US","location":[-97.822,37.751],"longitude":-97.822},"offset":250,
"macId":"ec:1f:72:b3:86:bb","input_type":"log","message":"10.108.2.44 \"73.112.77.205, 10.10.10.161\" - - [26/Jul/2017:00:00:00 -0500] \"GE
T /wod/landing?c=e&macId=ec%3A1f%3A72%3Ab3%3A86%3Abb&location=WODTrial&apMacId=c4%3A27%3A95%3A56%3Ad2%3Ae3&a=as&bn=st22&wod=1&issuer=r&devi
ceModel=default HTTP/1.1\" 500 -\n","kvstring":"macId=ec%3A1f%3A72%3Ab3%3A86%3Abb&location=WODTrial&apMacId=c4%3A27%3A95%3A56%3Ad2%3Ae3&a=a
s&bn=st22&wod=1&issuer=r&deviceModel=default","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.161","httpstatuscode":"
500","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.112.77.205","httpversion":"1.1","location":"WODTrial","deviceModel":"defaul
t","page":"/wod/landing?c=e"}{"geoip":{"timezone":"America/New_York","ip":"69.180.99.63","latitude":26.8457,"continent_code":"NA","city_nam
e":"Palm Beach Gardens","country_code2":"US","country_name":"United States","dma_code":548,"country_code3":"US","region_name":"Florida","lo
cation":[-80.0902,26.8457],"postal_code":"33410","longitude":-80.0902,"region_code":"FL"},"offset":173,"loadbalancerip":"10.108.2.44","inpu
t_type":"log","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access2.log","message":"10.108.2.44 \"69.180.99.63, 10.10.10.153\"
- - [26/Jul/2017:00:00:00 -0500] \"GET /widgetstore-client/businesswidget/repository/common/lib/requirejs/i18n.js HTTP/1.1\" 200 2382","typ
e":"omniwebaccss","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.153","httpstatuscode":"200","@timestamp":"2017-07-2
5T21:00:00.000Z","clientrealip":"69.180.99.63","httppayload":"2382","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","versi
on":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-80.0902,26.8457","httpversion":"1.1","page":"/widgetstore-client/businesswidget/repos
itory/common/lib/requirejs/i18n.js","timestamp":"26/Jul/2017:00:00:00"}{"loadbalancerip":"10.108.2.44","source":"/home/vpwrk1/ElasticDataFo
rTest/COMCAST/Res/app/access3.log","type":"omniwebaccss","bn":"st22","wod":"1","issuer":"r","@version":"1","beat":{"hostname":"vpelastic","
name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-95.4692,30.2264","apMacId":"10:86:8c:87:73:6a","timestamp":"2
6/Jul/2017:00:00:00","a":"ho","geoip":{"timezone":"America/Chicago","ip":"98.200.8.132","latitude":30.2264,"continent_code":"NA","city_name":"Conroe","country_code2":"US","country_name":"United States","dma_code":618,"country_code3":"US","region_name":"Texas","location":[-95.4692,30.2264],"postal_code":"77384","longitude":-95.4692,"region_code":"TX"},"offset":273,"macId":"ec:9b:f3:14:50:76","input_type":"log","message":"10.108.2.44 \"98.200.8.132, 10.10.10.164\" - - [26/Jul/2017:00:00:00 -0500] \"GET /wod/landing?c=e&macId=ec%3A9b%3Af3%3A14%3A50%3A76&location=WODTrial&apMacId=10%3A86%3A8c%3A87%3A73%3A6a&a=ho&bn=st22&wod=1&issuer=r&deviceModel=Linux+-+Ubuntu+Firefox+-+Linux HTTP/1.1\" 500 -\n\n","kvstring":"macId=ec%3A9b%3Af3%3A14%3A50%3A76&location=WODTrial&apMacId=10%3A86%3A8c%3A87%3A73%3A6a&a=ho&bn=st22&wod=1&issuer=r&deviceModel=Linux+-+Ubuntu+Firefox+-+Linux","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.164","httpstatuscode":"500","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"98.200.8.132","httpversion":"1.1","location":"WODTrial","deviceModel":"Linux+-+Ubuntu+Firefox+-+Linux","page":"/wod/landing?c=e"}
27

{"loadbalancerip":"10.108.2.44","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access4.log","type":"omniwebaccss","bn":"st22","wod":"1","issuer":"r","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-93.902,32.403","apMacId":"10:56:11:15:d3:8d","timestamp":"26/Jul/2017:00:00:00","a":"as","geoip":{"timezone":"America/Chicago","ip":"73.91.231.150","latitude":32.403,"continent_code":"NA","city_name":"Shreveport","country_code2":"US","country_name":"United States","dma_code":612,"country_code3":"US","region_name":"Louisiana","location":[-93.902,32.403],"postal_code":"71129","longitude":-93.902,"region_code":"LA"},"offset":258,"macId":"90:c7:d8:fa:78:e9","input_type":"log","message":"10.108.2.44 "73.91.231.150, 10.10.10.165" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/landing?c=e&macId=90%3Ac7%3Ad8%3Afa%3A78%3Ae9&location=WODTrial&apMacId=10%3A56%3A11%3A15%3Ad3%3A8d&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Jasper+LTE HTTP/1.1" 303 -\n\n","kvstring":"macId=90%3Ac7%3Ad8%3Afa%3A78%3Ae9&location=WODTrial&apMacId=10%3A56%3A11%3A15%3Ad3%3A8d&a=as&bn=st22&wod=1&issuer=r&deviceModel=ZTE+Jasper+LTE","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.165","httpstatuscode":"303","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.91.231.150","httpversion":"1.1","location":"WODTrial","deviceModel":"ZTE+Jasper+LTE","page":"/wod/landing?c=e"}{"loadbalancerip":"10.108.2.44","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access5.log","type":"omniwebaccss","bn":"st22","wod":"1","issuer":"r","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-89.7726,35.1364","apMacId":"3c:7a:8a:3f:5d:de","timestamp":"26/Jul/2017:00:00:00","a":"as","geoip":{"timezone":"America/Chicago","ip":"73.91.227.213","latitude":35.1364,"continent_code":"NA","city_name":"Cordova","country_code2":"US","country_name":"United States","dma_code":640,"country_code3":"US","region_name":"Tennessee","location":[-89.7726,35.1364],"postal_code":"38018","longitude":-89.7726,"region_code":"TN"},"offset":249,"macId":"60:a4:d0:5c:80:63","input_type":"log","message":"10.108.2.44 "73.91.227.213, 10.10.10.162" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/landing?c=e&macId=60%3Aa4%3Ad0%3A5c%3A80%3A63&location=WODTrial&apMacId=3c%3A7a%3A8a%3A3f%3A5d%3Ade&a=as&bn=st22&wod=1&issuer=r&deviceModel=default HTTP/1.1" 303 -","kvstring":"macId=60%3Aa4%3Ad0%3A5c%3A80%3A63&location=WODTrial&apMacId=3c%3A7a%3A8a%3A3f%3A5d%3Ade&a=as&bn=st22&wod=1&issuer=r&deviceModel=default","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.162","httpstatuscode":"303","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.91.227.213","httpversion":"1.1","location":"WODTrial","deviceModel":"default","page":"/wod/landing?c=e"}{"geoip":{"timezone":"America/Chicago","ip":"73.91.225.191","latitude":35.1364,"continent_code":"NA","city_name":"Cordova","country_code2":"US","country_name":"United States","dma_code":640,"country_code3":"US","region_name":"Tennessee","location":[-89.7726,35.1364],"postal_code":"38018","longitude":-89.7726,"region_code":"TN"},"offset":140,"loadbalancerip":"10.108.2.44","input_type":"log","source":"/home/vpwrk1/ElasticDataForTest/COMCAST/Res/app/access6.log","message":"10.108.2.44 "73.91.225.191, 10.10.10.162" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/static/res/css/captive-network.css HTTP/1.1" 200 20690","type":"omniwebaccss","tags":["beats_input_codec_plain_applied"],"clientnattedip":"10.10.10.162","httpstatuscode":"200","@timestamp":"2017-07-25T21:00:00.000Z","clientrealip":"73.91.225.191","httppayload":"20690","@version":"1","beat":{"hostname":"vpelastic","name":"vpelastic","version":"5.4.1"},"host":"vpelastic","clientGeoLocation":"-89.7726,35.1364","httpversion":"1.1","page":"/wod/static/res/css/captive-network.css","timestamp":"26/Jul/2017:00:00:00"}

28

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.