Logstash 2.3 - GeoIP problem

Hello people.. sorry to bother , once again

ALL of my config is working and I mean it..
absolutely all of it. mostely ripped code from siemonster and other places, but all together " work "

EXCEPT GeoIP

Please gimme a hand, heres the config :

filter  {
  if [type] == "syslog" {
      if "devname" in [message] {
      mutate {
      add_tag => [ "COUPEFEU", "FORTIGATE" ]
      }
        }

      if "%ASA-" in [message] {
      mutate {
      add_tag => [ "Firewall", "ASA" ]
      }
	  }
      if "VPN" in [message] {
      mutate {
      add_tag => [ "VPN" ]
      }
	}
	  if "SOC" in [message] {
      mutate {
      add_tag => [ "SOC" ]
      }
	} 
      if "IPS" in [message] {
      mutate {
      add_tag => [ "IPS" ]
      }
           }
      if "printer" in [message] {
      mutate {
      add_tag => [ "hp-printers" ]
           }
      }

#
#
#
# Parse Fortigate
if "FORTIGATE" in [tags] {
grok {
  match => [ "message", "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{GREEDYDATA:kv}" ]
  remove_field => ["message"]
  remove_field => ["syslog_timestamp"]
#  remove_field => ["type"]
}
syslog_pri { }

kv {
      source => "kv"
      exclude_keys => [ "type", "subtype" ]
      field_split => " "
      value_split => "="
}

date {
  match => [ "logtimestamp", "ISO8601" ]
  locale => "en"
  timezone =>"America/Montreal"
  remove_field => [ "logtimestamp" ]
}

mutate {
      convert => [ "rcvdbyte", "integer" ]
      convert => [ "countdlp", "integer" ]
      convert => [ "countweb", "integer" ]
      convert => [ "countav", "integer" ]
      convert => [ "countemail", "integer" ]
      convert => [ "countips", "integer" ]
      convert => [ "duration", "integer" ]
      convert => [ "sentpkt", "integer" ]
      convert => [ "rcvdpkt", "integer" ]
      convert => [ "sentbyte", "integer" ]
      convert => [ "shaperdroprcvdbyte", "integer" ]
      convert => [ "shaperdropsentbyte", "integer" ]
      convert => [ "filesize", "integer" ]
      convert => [ "count", "integer" ]
      convert => [ "total", "integer" ]
      convert => [ "totalsession", "integer" ]
      convert => [ "bandwidth", "integer" ]
      #rename => { "type" => "ftg-type" }
    }


#Geolocate logs that have SourceAddress and if that SourceAddress is a non-RFC1918 address or APIPA address
if [srcip] and [srcip] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    geoip {
         database => "/etc/logstash/GeoLiteCity.dat"
         source => "srcip"
         target => "SourceGeo"
         add_tag => [ "traffic-wan" ]
    }
}

#filtrer le traffic RITM du traffic internet pour input dans un second index
if [srcip] and [srcip] =~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    mutate {
         add_tag => [ "src-traffic-ritm" ]
         ["SourceGeo.location"] => "geo_point"
    }

    #Delete 0,0 in SourceGeo.location if equal to 0,0
    #if ([srcip.location] and [srcip.location] =~ "0,0") {
      #mutate {
       # ["SourceGeo.location"] => "geo_point"
      #}
    #}
  #}

#Geolocate logs that have DestinationAddress and if that DestinationAddress is a non-RFC1918 address or APIPA address
if [dstip] and [dstip] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    geoip {
         database => "/etc/logstash/GeoLiteCity.dat"
         source => "dstip"
         target => "DestinationGeo"
         add_tag => [ "traffic-wan" ]
    }
}

#filtrer le traffic RITM du traffic internet pour input dans un second index
if [dstip] and [dstip] =~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    mutate {
         add_tag => [ "dst-traffic-ritm" ]
         ["DestinationGeo.location"] => "geo_point"
    }

    #Delete 0,0 in DestinationGeo.location if equal to 0,0
    #if ([dstip.location] and [dstip.location] =~ "0,0") {
      #mutate {
       # ["DestinationGeo.location"] => "geo_point"
        # }
       #}
     }
  }
#
#
#
#

STRIPPED LACK OF SPACE - pastebin here : https://pastebin.com/4DnmxDa1

somehow, it just.. doesnt populate the geoip.. dont know why nor where to look for an answer...
Thank you .. and sorry to bother :\

Please show an example event, preferably using a stdout { codec => rubydebug } output.

Hello

heres an exemple. please see pastebin :

https://pastebin.com/1rcjH2jS

Thank you.

And what isn't working? The DestinationGeo field appears to be populated just fine. The SourceGeo isn't, but that's expected with the 10.0.0.0/8 address in srcip.

Im not at the office right now and tomorrow but I just cant get a tile map to work.

Theres no "field" geo

You mean Kibana doesn't list any geo_point fields to build your tile map from? If so you'll have to adjust your index template so that additional fields have the geo_point. Read up on mappings and index templates in the ES documentation.

Geo.point is present on the kibana visualisation
Yet it is empty

I don't know what you mean. Show a screenshot and what the index's mappings look like (use ES's get mapping API).

Heres the screenshot as requested.

and the mapping :

{
"order": 0,
"template": "logstash-",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
},
{
"string_fields": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "
"
}
}
],
"_all": {
"omit_norms": true,
"enabled": true
},
"properties": {
"@timestamp": {
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
}
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {}
}

Ive just modified it to match ... ( at least I think ) we will see tomorrow on index-creation

{
  "order": 0,
  "template": "logstash-*",
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        {
          "message_field": {
            "mapping": {
              "fielddata": {
                "format": "disabled"
              },
              "index": "analyzed",
              "omit_norms": true,
              "type": "string"
            },
            "match_mapping_type": "string",
            "match": "message"
          }
        },
        {
          "string_fields": {
            "mapping": {
              "fielddata": {
                "format": "disabled"
              },
              "index": "analyzed",
              "omit_norms": true,
              "type": "string",
              "fields": {
                "raw": {
                  "ignore_above": 256,
                  "index": "not_analyzed",
                  "type": "string"
                }
              }
            },
            "match_mapping_type": "string",
            "match": "*"
          }
        }
      ],
      "_all": {
        "omit_norms": true,
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "geoip": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "DestinationGeo": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "SourceGeo": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "@version": {
          "index": "not_analyzed",
          "type": "string"
        }
      }
    }
  },
  "aliases": {}
}

unfortunately ..

still not working :\

dang

Any help please ? im sure its something really simple... probably with my mapping..

Please show a) an example document and b) the index mappings (not the index template).

Elastic HQ :

mapping : https://pastebin.com/2v87MK7H

Okay, so both [SourceGeo][location] and [DestinationGeo][location] have been mapped as geo_point. Are those field still not available in Kibana?

Negative.

They are not in kabana.

Did you refresh the field list in Kibana?

Yes I did.
Right now I am not at the office. But tomorrow in about 6-7h ill be

Ill be able to post another screenshot of the issue.

image

heres the "field" in kibana when trying to visualize

Please show:

  • An example document. Copy/paste from Kibana's JSON tab.
  • The index mappings. Use the get mapping API. No screenshot.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.