Missing geo.location

[Jan_Kaspar (Jan)]

Hi all,

I would like to ask for help. I am parsing data from our firewall and there are fields containing IP address. I would like to use Coordinates map to visualize accesses. Problem is that i am missing geo.location field in parsed data.

ELK 7.2.0

logstash config:

filter {
if "USG3P" in [tags] {
grok {
match => {
"[message]" => [
"%{TIMESTAMP_ISO8601:date} %{DATA:device} %{DATA:logsource}:\s+[%{DATA:rules.int}-%{DATA:rules.index}-%{DATA:rules.action}]IN=%{DATA:interfaces.input}\s+OUT=%{DATA:interfaces.output}\s+MAC=%{DATA:mac}\s+SRC=%{IPV4:SRC}\s+DST=%{IPV4:DST}\s+LEN=%{INT:LEN}\s+TOS=%{BASE16NUM:TOS}\s+PREC=%{BASE16NUM:PREC}\s+TTL=%{INT:TTL}\s+ID=%{INT:ID}\s+%{DATA:flags}\s+PROTO=%{WORD:PROTO}\s+SPT=%{INT:SPT}\s+DPT=%{INT:DPT}\s+WINDOW=%{INT:WINDOW}\s+RES=%{BASE16NUM:RES}\s+%{WORD:flag1}\s+%{WORD:flag2}\s+%{WORD:flag3}\s+URGP=%{WORD:URGP}%{GREEDYDATA:rest}",
"%{TIMESTAMP_ISO8601:date} %{DATA:device} %{DATA:logsource}:\s+[%{DATA:rules.int}-%{DATA:rules.index}-%{DATA:rules.action}]IN=%{DATA:interfaces.input}\s+OUT=%{DATA:interfaces.output}\s+MAC=%{DATA:mac}\s+SRC=%{IPV4:SRC}\s+DST=%{IPV4:DST}\s+LEN=%{INT:LEN}\s+TOS=%{BASE16NUM:TOS}\s+PREC=%{BASE16NUM:PREC}\s+TTL=%{INT:TTL}\s+ID=%{INT:ID}\s+%{DATA:flags}\s+PROTO=%{WORD:PROTO}\s+SPT=%{INT:SPT}\s+DPT=%{INT:DPT}\s+WINDOW=%{INT:WINDOW}\s+RES=%{BASE16NUM:RES}\s+%{WORD:flag1}\s+URGP=%{WORD:URGP}%{GREEDYDATA:rest}"
]
}
}
geoip {
source => "SRC"
}
}
}

here is index pattern:

detail of parsed fields: (missing geo.location)

Any suggestion?

Thank you Jan

[Badger]

Do you have a template similar to the default logstash-* template that tells elasticsearch that geoip.location should be a geo_point in the filebeat-* indexes?

[Jan_Kaspar (Jan)]

Hi Badger,

thanks for answer. Look at my first screenshot. there is geo.location. I already tryed to use that. but it didnt worked.

My be I am using it wrong because I am getting this error:

"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [default : {dynamic_templates=[{message_field={path_match=message, mapping={norms=false, type=text}, match_mapping_type=string}}, {string_fields={mapping={norms=false, type=text, fields={keyword={ignore_above=256, type=keyword}}}, match_mapping_type=string, match=}}], properties={@timestamp={type=date}, geoip={dynamic=true, properties={ip={type=ip}, latitude={type=half_float}, location={type=geo_point}, longitude={type=half_float}}}, @version={type=keyword}}}]"
}
],
"type" : "mapper_parsing_exception",
"reason" : "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [default : {dynamic_templates=[{message_field={path_match=message, mapping={norms=false, type=text}, match_mapping_type=string}}, {string_fields={mapping={norms=false, type=text, fields={keyword={ignore_above=256, type=keyword}}}, match_mapping_type=string, match=}}], properties={@timestamp={type=date}, geoip={dynamic=true, properties={ip={type=ip}, latitude={type=half_float}, location={type=geo_point}, longitude={type=half_float}}}, @version={type=keyword}}}]",
"caused_by" : {
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [default : {dynamic_templates=[{message_field={path_match=message, mapping={norms=false, type=text}, match_mapping_type=string}}, {string_fields={mapping={norms=false, type=text, fields={keyword={ignore_above=256, type=keyword}}}, match_mapping_type=string, match=*}}], properties={@timestamp={type=date}, geoip={dynamic=true, properties={ip={type=ip}, latitude={type=half_float}, location={type=geo_point}, longitude={type=half_float}}}, @version={type=keyword}}}]"
}
},
"status" : 400
}

Jan

[Jan_Kaspar (Jan)]

Solved with this finaly:

curl -X PUT "localhost:9200/_template/filebeat-7.2.0?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["filebeat-"],
"settings": {
"number_of_shards": 1
},
"mappings" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
'