Illegal latitude value exception in elasticsearch

Elastic Stack Elasticsearch
1

Hi,

I am getting the following exception in Elasticsearch:

Caused by: java.lang.IllegalArgumentException:illegal latitude value [269.9999642930925] for clientrealip
at
clientrealip is my ip filed.

In the logstash output:

"clientrealip" => "73.35.67.125"

I tried to convert to fload, but it doesn't help.

            geoip {
                   source => "clientrealip"
                   target => "geoip.location"
                   #database => "/etc/logstash/GeoLiteCity.dat"
                   add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][longitude]}" ]
                   add_field => [ "[geoip.location][coordinates]", "%{[geoip.location][latitude]}" ]
            }
            mutate
            {
                convert => { "[geoip.location][latitude]" => "float" }
            }

The output in logstash also show :

   "geoip.location" => {
              "timezone" => "America/New_York",
                    "ip" => "73.35.67.125",
              "latitude" => 29.7596,
        "continent_code" => "NA",
             "city_name" => "St. Augustine",
         "country_code2" => "US",
          "country_name" => "United States",
              "dma_code" => 561,
         "country_code3" => "US",
           "region_name" => "Florida",
              "location" => [
            [0] -81.3029,
            [1] 29.7596
        ],
           "postal_code" => "32086",
             "longitude" => -81.3029,
           "region_code" => "FL"
    },

What am I missing here? How can I solve it?

Regards,
Sharon.

2

What is your mapping?

3 
PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "geo_point"},
          .....
          ......
4

I had also that in the mapping:

"@timestamp":{"type":"date"},

But it cause an exception in the Elastic, So I took it out from the mapping.

5

clientrealip Is an IP not a geo point.

6

But if I want to be able to create a map visualization, I need to set a geo point. Not?
Where will I set a geo point?

7

No. An IP can't be a geo point.

Probably geoip.location.location field is a geo point.

8

Hi,
The answer isn't clear.
What can be a geo point?
Which field do I need to use as source?

9

geoip.location.location can be a geo point.

Which field do I need to use as source?

Not sure I understand but basically you should write a config like:

            geoip {
                   source => "clientrealip"
            }

Without anything else.

Then define in your mapping geoip.location as a geopoint.

10

great.
Will do it right now.

Thanks

11

Unbelievable...
Whatever I am trying, it doesn't work. I already spent on it days....
I did exactly as you wrote, but still can't make it work.

How do I define in my mapping the geoip.location as a geopint?

Like that:
"geoip": {"type": "geo_point"}

Then I am getting the geoip.location as a number :

location
geoip.location.jpg1047×535 106 KB

or like that?

"geoip.location": {"type": "geo_point"}

Then I am getting the following error:

earch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: org.elasticsearch.ElasticsearchParseException: geo_point expected
        at org.elasticsearch.common.geo.GeoUtils.parseGeoPoint(GeoUtils.java:465) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:581) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5
12

Hi,

Can anyone help here?

It is super urgent.

Thanks
Sharon.

13

Please read About the Elasticsearch category. Specifically the be patient part.

I'm planning to answer you but not immediately sorry.

14

OK. Thanks.

15

This is working well:

DELETE test
PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}
PUT test/doc/1
{
  "location": [ -81.3029, 29.7596 ]
}
16

Thanks for your time.

My mapping looks different:

PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "ip"},
         "loadbalancerip":{"type": "ip"},
         "clientnattedip":{"type": "ip"},
         "extrnalip":{"type": "ip"},
         "method":{"type": "keyword", "index": true},
         "page":{"type": "keyword", "index": true},
         "c":{"type": "keyword", "index": true},
         "macid":{"type": "keyword", "index": true},
         "bn":{"type": "keyword", "index": true},
         "a":{"type": "keyword", "index": true},
         "location":{"type": "keyword", "index": true},
         "apmacid":{"type": "keyword", "index": true},
         "wodflag":{"type": "keyword", "index": true},
         "wod":{"type": "keyword", "index": true},
         "issuer":{"type": "keyword", "index": true},
         "accesstoken":{"type": "keyword", "index": true},
         "devicename":{"type": "keyword", "index": true},
         "devicemodel":{"type": "text", "index": true},
         "httpversion":{"type": "float"},
         "httpstatuscode":{"type": "integer"},
         "httppayload":{"type": "integer"},
         "description":{"type": "keyword", "index": true},
         "geoip": {"type": "geo_point"}
      }
    }
  }
}

Does it look for you problematic?

17

Yes it is.

         "geoip": {"type": "geo_point"}

This can't work. As the datastructure you are generating is probably like:

   "geoip" => {
              "timezone" => "America/New_York",
                    "ip" => "73.35.67.125",
              "latitude" => 29.7596,
        "continent_code" => "NA",
             "city_name" => "St. Augustine",
         "country_code2" => "US",
          "country_name" => "United States",
              "dma_code" => 561,
         "country_code3" => "US",
           "region_name" => "Florida",
              "location" => [
            [0] -81.3029,
            [1] 29.7596
        ],
           "postal_code" => "32086",
             "longitude" => -81.3029,
           "region_code" => "FL"
    },

As I said earlier, geoip.location should be the geo_point type field.

18

Still not working.

If I understood you right, So I did like that:

1. I changed the field name that was we called it "location" to "msglocation" in the grok and in the mapping. This field is a string.

2. I added to the the mapping the location as you advice.

So My final mapping looks like that:

PUT _template/omniwebaccss_template
{
  "template": "logstash*omniwebaccss",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "geo_point_type": {
      "properties": {
         "@version":{"type": "keyword", "index": true},
         "clientrealip": {"type": "ip"},
         "loadbalancerip":{"type": "ip"},
         "clientnattedip":{"type": "ip"},
         "extrnalip":{"type": "ip"},
         "method":{"type": "keyword", "index": true},
         "page":{"type": "keyword", "index": true},
         "c":{"type": "keyword", "index": true},
         "macid":{"type": "keyword", "index": true},
         "bn":{"type": "keyword", "index": true},
         "a":{"type": "keyword", "index": true},
         "msglocation":{"type": "keyword", "index": true},
         "apmacid":{"type": "keyword", "index": true},
         "wodflag":{"type": "keyword", "index": true},
         "wod":{"type": "keyword", "index": true},
         "issuer":{"type": "keyword", "index": true},
         "accesstoken":{"type": "keyword", "index": true},
         "devicename":{"type": "keyword", "index": true},
         "devicemodel":{"type": "text", "index": true},
         "httpversion":{"type": "float"},
         "httpstatuscode":{"type": "integer"},
         "httppayload":{"type": "integer"},
         "description":{"type": "keyword", "index": true},
         "location": { "type": "geo_point"}
      }
    }
  }
}

I am getting the following exception now:

earch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.1.jar:5.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.lang.IllegalArgumentException: illegal latitude value [269.99982833862305] for location
        at org.elasticsearch.index.mapper.LatLonPointFieldMapper.parse(LatLonPointFieldMapper.java:160) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parsePointFromString(BaseGeoPointFieldMapper.java:607) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:578) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450) ~[elasticsearch-5.4.1.jar:5.4.1]

Any idea? How to continue?

19

What does a document look like?

Also what is the real mapping? Not the template.

20

The document looks like:

10.108.2.44 "73.112.77.205, 10.10.10.161" - - [26/Jul/2017:00:00:00 -0500] "GET /wod/landing?c=e&macId=ec%3A1f%3A72%3Ab3%3A86%3Abb&location=WODTrial&apMacId=c4%3A27%3A95%3A56%3Ad2%3Ae3&a=as&bn=st22&wod=1&issuer=r&deviceModel=default HTTP/1.1" 500 -

Real mapping, you mean the grok?

                       message => [
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{LOCALSTRING:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{LOCALSTRING:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{GREEDYDATA:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{DATA}\=%{GREEDYDATA:msglocation}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{GREEDYDATA:a}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{LOCALSTRING:wod}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{WORD:method} %{DATA:page}\?%{WORD}\=%{DATA:c}\&%{WORD}\=%{URLMAC:macid}\&%{WORD}\=%{LOCALSTRING:bn}\&%{WORD}\=%{GREEDYDATA:a}\&%{DATA}\=%{URLMAC:apmacid}\&%{WORD}\=%{LOCALSTRING:issuer}\&%{WORD}\=%{DATA:wodflag}\&%{WORD}\=%{LOCALSTRING:msglocation}\&%{WORD}\=%{GREEDYDATA:accesstoken}\&%{WORD}\=%{LOCALSTRING:devicename}\&%{WORD}\=%{GREEDYDATA:devicemodel} HTTP/%{NUMBER:httpversion}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?",
                            "%{IP:loadbalancerip} \"-\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}",
                            "%{IP:loadbalancerip} \"%{IP:clientrealip}, %{IP:clientnattedip}, %{IP:extrnalip}\" - - \[%{HTTPDATE:timestamp} %{INT}\] \"%{GREEDYDATA:description}\" %{BASE10NUM:httpstatuscode} %{BASE10NUM:httppayload}?"
                       ]

*It include private patterns.