How to check if the field exists before to define him as geoppoint

Elastic Stack Logstash
1

Hello,

I use Filebeats, Logstash and Elasticsearch to parse differents types of logs files ( IIS, Syslog, Apache, ... ) and watch the result with Kibana. I want to add a map on my dashboard so I use geoip to get gps coordinates from an ip address but sometimes (to set an example : with syslog log file) I only have the username and not the ip address. In result only a part of my output data have the geoip field but i want to define geoip.location as a geopoint in my template. Can i just set the type of the field:"location" on geo_point even if the field is not common to all the output ?

Thank you in advance for your answer

2

Having a template that defines the type of a field does not mean any documents have to contain that field, so you can set the type of an optional field.

3

Perfect I thank you ! But my problem isn't solved :disappointed_relieved: i asked this question because when i try to convert geoip.location into a geo_point it doesn't work ( i tried to find the issue without sucess). I followed this tutorial (and others )https://www.elastic.co/fr/blog/geoip-in-the-elastic-stack
i actually use geoip, but if i can't see the location datas on a map, use it seems meaningless to me
Do you have any solutions ?

4

Before applying the geoip filter on the field xyz, check the existence of that field, like:

filter {
  if [xyz] {
    geoip { source => "xyz" }
  }
}
5

Thank you but I already did it. I check the existence of the field and his type ( an ip address ) before to apply geoip.
In fact when i set logstash output on stdout{} i get all the information that i want ( like the geoip field ), but when i set logstash output on elacticsearch and use a tempate to define the field as a geo_point, elasticsearch save it as a number and not a geo_point so i can't use the map visualization in kibana. This is my current issue.

6

Oh got it now.

Can you share the template? Are you sure it's been applied to your index (have you created it BEFORE creating the index)?

7

Sorry i'm late
In logstash

geoip {
source => "clientip"
target => "geoip"
}
mutate {
convert => {
"[geoip][location][lat]" => "float"
"[geoip][location][lon]" => "float"
}
}

Mutate part is only to be sure that logstash save location as float and not a number
My template:

{
"index_patterns": ["squid-*"],
"mappings": {
"default": {
"properties": {
"clientip": { "type": "ip" },
"server_ip": { "type": "ip" },
"user": { "type": "keyword" },
"timestamp": { "type": "date" },
"Log_File_Format": { "type": "keyword" },
"Data_Sent": { "type": "number" },
"Data_Received": { "type": "number" },
"HTTP_Result": { "type": "number" },
"bytes": { "type": "number" },
"delay": { "type": "number" },
"facility": { "type": "number" },
"httpversion": { "type": "number" },
"pid": { "type": "number" },
"priority": { "type": "number" },
"response": { "type": "number" },
"windows_response": { "type": "number" },
"geoip": {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location": { "type": "geo_point" }
}
}
}
}
}
}

I tried a lot of things : create a new field and put lat and lon inside , convert all what i can into float type and check the type of all the fields,...
but it doesn't work
I think i will create a new topic because you answered my first question ( and i thank you ) and it's an other problem

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.