Illegal latitude value exception in elasticsearch


(Sharon Sasporta) #1

Hi,

I am getting the following exception in Elasticsearch:

Caused by: java.lang.IllegalArgumentException:illegal latitude value [269.9999642930925] for clientrealip
at
clientrealip is my ip filed.

In the logstash output:

"clientrealip" => "73.35.67.125"

I tried to convert to fload, but it doesn't help.

            geoip {
                   source => "clientrealip"
                   target => "geoip.location"
                   #database => "/etc/logstash/GeoLiteCity.dat"
                   add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][longitude]}" ]
                   add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][latitude]}" ]
            }
            mutate
            {
                convert => { "[geoip.location][latitude]" => "float" }
            } 

The output in logstash also show :

   "geoip.location" => {
              "timezone" => "America/New_York",
                    "ip" => "73.35.67.125",
              "latitude" => 29.7596,
        "continent_code" => "NA",
             "city_name" => "St. Augustine",
         "country_code2" => "US",
          "country_name" => "United States",
              "dma_code" => 561,
         "country_code3" => "US",
           "region_name" => "Florida",
              "location" => [
            [0] -81.3029,
            [1] 29.7596
        ],
           "postal_code" => "32086",
             "longitude" => -81.3029,
           "region_code" => "FL"
    },

What am I missing here? How can I solve it?

Regards,
Sharon.


(David Pilato) #2

What is your mapping?


(Sharon Sasporta) #3
PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "geo_point"},
          .....
          ......

(Sharon Sasporta) #4

I had also that in the mapping:

"@timestamp":{"type":"date"},

But it cause an exception in the Elastic, So I took it out from the mapping.


(David Pilato) #5

clientrealip Is an IP not a geo point.


(Sharon Sasporta) #6

But if I want to be able to create a map visualization, I need to set a geo point. Not?
Where will I set a geo point?


(David Pilato) #7

No. An IP can't be a geo point.

Probably geoip.location.location field is a geo point.


(Sharon Sasporta) #8

Hi,
The answer isn't clear.
What can be a geo point?
Which field do I need to use as source?


(David Pilato) #9

geoip.location.location can be a geo point.

Which field do I need to use as source?

Not sure I understand but basically you should write a config like:

            geoip {
                   source => "clientrealip"
            }

Without anything else.

Then define in your mapping geoip.location as a geopoint.


(Sharon Sasporta) #10

great.
Will do it right now.

Thanks


(Sharon Sasporta) #11

Unbelievable...
Whatever I am trying, it doesn't work. I already spent on it days....
I did exactly as you wrote, but still can't make it work.

How do I define in my mapping the geoip.location as a geopint?

Like that:
"geoip": {"type": "geo_point"}

Then I am getting the geoip.location as a number :

or like that?

"geoip.location": {"type": "geo_point"}

Then I am getting the following error:

earch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: org.elasticsearch.ElasticsearchParseException: geo_point expected
        at org.elasticsearch.common.geo.GeoUtils.parseGeoPoint(GeoUtils.java:465) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:581) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5

(Sharon Sasporta) #12

Hi,

Can anyone help here?

It is super urgent.

Thanks
Sharon.


(David Pilato) #13

Please read About the Elasticsearch category. Specifically the be patient part.

I'm planning to answer you but not immediately sorry.


(Sharon Sasporta) #14

OK. Thanks.


(David Pilato) #15

This is working well:

DELETE test
PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}
PUT test/doc/1
{
  "location": [ -81.3029, 29.7596 ]
}

(Sharon Sasporta) #16

Thanks for your time.

My mapping looks different:

PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "ip"},
         "loadbalancerip":{"type": "ip"},
         "clientnattedip":{"type": "ip"},
         "extrnalip":{"type": "ip"},
         "method":{"type": "keyword", "index": true},
         "page":{"type": "keyword", "index": true},
         "c":{"type": "keyword", "index": true},
         "macid":{"type": "keyword", "index": true},
         "bn":{"type": "keyword", "index": true},
         "a":{"type": "keyword", "index": true},
         "location":{"type": "keyword", "index": true},
         "apmacid":{"type": "keyword", "index": true},
         "wodflag":{"type": "keyword", "index": true},
         "wod":{"type": "keyword", "index": true},
         "issuer":{"type": "keyword", "index": true},
         "accesstoken":{"type": "keyword", "index": true},
         "devicename":{"type": "keyword", "index": true},
         "devicemodel":{"type": "text", "index": true},
         "httpversion":{"type": "float"},
         "httpstatuscode":{"type": "integer"},
         "httppayload":{"type": "integer"},
         "description":{"type": "keyword", "index": true},
         "geoip": {"type": "geo_point"}
      }
    }
  }
}

Does it look for you problematic?


(David Pilato) #17

Yes it is.

         "geoip": {"type": "geo_point"}

This can't work. As the datastructure you are generating is probably like:

   "geoip" => {
              "timezone" => "America/New_York",
                    "ip" => "73.35.67.125",
              "latitude" => 29.7596,
        "continent_code" => "NA",
             "city_name" => "St. Augustine",
         "country_code2" => "US",
          "country_name" => "United States",
              "dma_code" => 561,
         "country_code3" => "US",
           "region_name" => "Florida",
              "location" => [
            [0] -81.3029,
            [1] 29.7596
        ],
           "postal_code" => "32086",
             "longitude" => -81.3029,
           "region_code" => "FL"
    },

As I said earlier, geoip.location should be the geo_point type field.


(Sharon Sasporta) #18

Still not working.

If I understood you right, So I did like that:

1. I changed the field name that was we called it "location" to "msglocation" in the grok and in the mapping. This field is a string.

2. I added to the the mapping the location as you advice.

So My final mapping looks like that:

PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "ip"},
         "loadbalancerip":{"type": "ip"},
         "clientnattedip":{"type": "ip"},
         "extrnalip":{"type": "ip"},
         "method":{"type": "keyword", "index": true},
         "page":{"type": "keyword", "index": true},
         "c":{"type": "keyword", "index": true},
         "macid":{"type": "keyword", "index": true},
         "bn":{"type": "keyword", "index": true},
         "a":{"type": "keyword", "index": true},
         "msglocation":{"type": "keyword", "index": true},
         "apmacid":{"type": "keyword", "index": true},
         "wodflag":{"type": "keyword", "index": true},
         "wod":{"type": "keyword", "index": true},
         "issuer":{"type": "keyword", "index": true},
         "accesstoken":{"type": "keyword", "index": true},
         "devicename":{"type": "keyword", "index": true},
         "devicemodel":{"type": "text", "index": true},
         "httpversion":{"type": "float"},
         "httpstatuscode":{"type": "integer"},
         "httppayload":{"type": "integer"},
         "description":{"type": "keyword", "index": true},
         "location": { "type": "geo_point"}
      }
    }
  }
}

I am getting the following exception now:

earch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.lang.IllegalArgumentException: illegal latitude value [269.99982833862305] for location
        at org.elasticsearch.index.mapper.LatLonPointFieldMapper.parse(LatLonPointFieldMapper.java:160) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parsePointFromString(BaseGeoPointFieldMapper.java:607) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:578) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]

Any idea? How to continue?


(David Pilato) #19

What does a document look like?

Also what is the real mapping? Not the template.


(Sharon Sasporta) #20

The document looks like:

10.108.2.44 "73.112.77.205, 10.10.10.161" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/landing?c=e&macId=ec%3A1f%3A72%3Ab3%3A86%3Abb&location=WODTrial&apMacId=c4%3A27%3A95%3A56%3Ad2%3Ae3&a=as&bn=st22&wod=1&issuer=r&deviceModel=default HTTP/1.1" 500 -

Real mapping, you mean the grok?

                       message => [
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{LOCALSTRING:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{LOCALSTRING:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{GREEDYDATA:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{GREEDYDATA:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{GREEDYDATA:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{GREEDYDATA:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?"
                       ]

*It include private patterns.