Geop.location not of geo_pooint. elastic.co 5.6.4 on RPI3

bruno-d · May 27, 2018, 7:42am

Hey, i did some change, renew my conf and my MAP is gone. No field of geo_point type.
I use the geoip pipeline filter and all is running fine except the map.

GET /logstash-2018.05/_mapping return a struct showing that geoip has a field name location with lat and lon. (see capture) they are filled correctly by the filter also.
I read many threads and in one that i could modify the logstash indice and delete the index then i will recreate. I apply the following
PUT _template/logstash
{
"template": "logstash-*",
"settings": {
"number_of_replicas": 1,
"number_of_shards": 1
},
"mappings": {
"syslog": {
"dynamic": "true",
"properties": {
"geoip": {
"dynamic": true,
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
}

and then delete the index pattern and recreate again. it failed.

Can someone help ?
thanks

dadoonet · May 27, 2018, 8:11am

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Could you share the full mapping?

bruno-d · May 27, 2018, 10:36am

Format error

dadoonet · May 28, 2018, 4:42am

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

bruno-d · May 28, 2018, 9:39am

Format errata

dadoonet · May 28, 2018, 9:59am

Did you check the preview window?

bruno-d · May 30, 2018, 1:14am

Hi, new try.

I try to format it as possible to show the geoip struct and location which should be the issue.
Just to relight the situation : Logstash index does not allow to be mapped as there's no geo_point field. GeoIP information are perfectly filled per the pipeline filter from the VarX vaiable which is the IP detected. It works fine except the map.

here is the GET /logstash-2018.05/_mapping

"logstash-2018.05":
    { "mappings": { 
    "syslog": { "properties": 
    { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "CPU": { "type": "long" }, "Disk": { "type": "long" }, "Port": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "RAM": { "type": "long" }, "Temp": { "type": "long" }, 
    "Varx": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } },

 "geoip": { "properties": { 
"city_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } },
 "continent_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } },
 "country_iso_code": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, 
"location": { "properties": { "lat": { "type": "float" }, "lon": { "type": "float" } } }, 
"region_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } 
}, 
"host": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "received_at": { "type": "date" }, "received_from": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_facility": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_facility_code": { "type": "long" }, "syslog_hostname": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_pid": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_program": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_severity": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "syslog_severity_code": { "type": "long" }, "syslog_timestamp": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "tags": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } } }` /&gt;

I try to

 `PUT _template/logstash
{
"template": "logstash-*",
"settings": {
"number_of_replicas": 1,
"number_of_shards": 1
},
"mappings": {
    "syslog": {
    "dynamic": "true",
    "properties": {
    "geoip": {
        "dynamic": true,
        "properties": {
            "location": {
                "type": "geo_point"
            }
        }
     }
}
}
}`

Then delete the index and recreate it.

dadoonet · May 30, 2018, 5:08am

Look

This is not formatted code

This is formatted code

Can you do this please? If not please explain what you don't understand.

bruno-d · May 30, 2018, 10:13pm

Oh yes got it now Sorry ! I corrected upper in the thread and removed the unformated messages. Thanks. Don't know how insert colors in the formatted like you did in red.

dadoonet · June 1, 2018, 6:56pm

I had to parse your post to make it readable.

{
   "mappings":{
      "syslog":{
         "properties":{
            "@timestamp":{
               "type":"date"
            },
            "@version":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "CPU":{
               "type":"long"
            },
            "Disk":{
               "type":"long"
            },
            "Port":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "RAM":{
               "type":"long"
            },
            "Temp":{
               "type":"long"
            },
            "Varx":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "geoip":{
               "properties":{
                  "city_name":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "continent_name":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "country_iso_code":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  },
                  "location":{
                     "properties":{
                        "lat":{
                           "type":"float"
                        },
                        "lon":{
                           "type":"float"
                        }
                     }
                  },
                  "region_name":{
                     "type":"text",
                     "fields":{
                        "keyword":{
                           "type":"keyword",
                           "ignore_above":256
                        }
                     }
                  }
               }
            },
            "host":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "message":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "received_at":{
               "type":"date"
            },
            "received_from":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_facility":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_facility_code":{
               "type":"long"
            },
            "syslog_hostname":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_message":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_pid":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_program":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_severity":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "syslog_severity_code":{
               "type":"long"
            },
            "syslog_timestamp":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "tags":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            },
            "type":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            }
         }
      }
   }
}

And:

PUT _template/logstash
{
   "template":"logstash-*",
   "settings":{
      "number_of_replicas":1,
      "number_of_shards":1
   },
   "mappings":{
      "syslog":{
         "dynamic":"true",
         "properties":{
            "geoip":{
               "dynamic":true,
               "properties":{
                  "location":{
                     "type":"geo_point"
                  }
               }
            }
         }
      }
   }
}

Just wondering why did you set "dynamic": true, in geoip field?

bruno-d · June 2, 2018, 4:26am

Hi, thanks how come it's nice with colors ? i will practice.
I put the dynamic to true because i found it in many thread on the subject. I have a poor level on Elastisearch but i am having fun. I have not been able to solve.

dadoonet · June 2, 2018, 7:56am

Pasting correct JSON (was missing curly braces or too many ones in your post), nice indentation... That's the key of the success.

I have not been able to solve.

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem.

bruno-d · June 2, 2018, 12:37pm

hi,

When I try to create a map of my logstash index I got the error :
No Compatible Fields: The "logstash-*" index pattern does not contain any of the following field types: geo_point.
I read many treads and found that geo_point is a [lat,long] struct with lat-long declared as numbers but it's not strictly declared as GeoPoint.
The GeoIP pipeline filter is running fine, filling all the field of the mentionned GeoIP structure ;
The thread describe the logstash index struct as well as a PUT procedure to try to register a GeoIP struct. cf. upper ref PUT _template/logstash

I am still unable to have a Map with lat,lon fields filled with correct value. and the message decribes in Step 1.

Regards

dadoonet · June 2, 2018, 1:06pm

Please share what you ran.

bruno-d · June 3, 2018, 12:09am

I ran it again to make sure of the process and it's working

Delete from the GUI Managment/ Index Pattern : Delete logstash-*
PUT _template/logstash according to the command line in the thread

Register VarX fied (may be not necessary as it is already in the datas)

 PUT /.kibana/_mapping/syslog
 {
  "properties": {
    "Varx": {
       "type": "text"
    }
   }  
}

I register the GeoIP pipeline filter to VarX variable (containing the IP address)

PUT _ingest/pipeline/geoip
{
  "description" : "Add geoip info",
 "processors" : [
  {
     "geoip" : {
                "field" : "Varx"
               }
    }]
}

I reconstruct the index from the GUI Managment / Index Pattern : Create Logstash-* and it works

I am so happy I have the MAP and GeoIP.location is of geo_point type in the Kibana/Managment/Index Pattern as in the screen capture.

system · July 1, 2018, 12:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.