Create point to point map from Logstash pipeline

Hello,

I am using logstash to parse my firewall logs and enrich them with geo location information

    geoip { source => "source.ip"
            target => "source.geo"
    }
    geoip { source => "destination.ip"
            target => "destination.geo"
     }
    geoip { source => "source.nat.ip"
            target => "source.nat.geo"
     }
   }

I want to create a point to point map using these informations, but when I click on the map I can't see my firewall-* index which contain my firewall logs

image

Could you please help me to understand why this is happening ?

Thanks

Didn't see the output index in the config.. Is there logstash output index pointed to firewall-*?
If yes, just create the index pattern based on the name firewall-*

I already have the index named firewall-*, the only problem is that it doesn't appear in the point to point map.

NB: The firewall-* index appear in the EMS Boundaries map

image

Hello, I've the same exact problem. Did you have any news on this topic?
Thanks Gianluca

Are the geopoints mapped correctly in the index?

@gian72 didn't find a solution yet

@warkolm, I am creating the index and the mapping like that:

the template:

PUT _template/firewall
{
  "index_patterns": ["firewall-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "hot-warm-cold-delete-6months-policy",  
    "index.lifecycle.rollover_alias": "firewall"
  },
  "mappings": {
    "properties": {
     "event": {
       "properties": {
         "severity": {
           "properties": {
             "code": { "type": "integer"},
             "name": { "type": "keyword"}
           }
         }
       }
     },
     "confid": {"type": "integer" },
     "slotlevel": {"type": "integer"},
     "rule": {
       "properties": {
         "id": { "type": "integer"},
         "name": {"type": "keyword" }
       }
     },
     "user": {
       "properties": {
         "name": {"type": "keyword"}
       }
     },
     "domain": {"type": "text"},
     "source": {
       "properties": {
         "interface_name": {"type": "keyword"},
         "ip":{ "type": "ip"},
         "ip_reputation": {"type": "keyword"},
         "port": {"type": "integer"},
         "port_name": {"type": "keyword" },
         "mac": {"type": "keyword"},
         "nat": {
           "properties":{
             "ip": {"type": "ip"},
             "port": {"type": "integer"}
           }
         }
       }
     },
     "srcifname": {"type": "keyword"},
     "network": {
       "properties": {
         "protocol": { "type": "keyword"}
       }
     },
     "destination": {
       "properties": {
         "interface_name": {"type": "keyword"},
         "ip": { "type": "ip"},
         "ip_reputation": {"type": "keyword"},
         "port": {"type": "integer"},
         "port_name": {"type": "keyword"}
       }
     },
     "dstifname": {"type": "text"},
     "icmp": {
       "properties": {
         "type": { "type": "integer"},
         "code": { "type": "integer"}
       }
     },
     "proto": {"type": "keyword"},
     "srcname": {"type": "keyword"},
    "src": {
      "properties": {
        "continent": {"type": "keyword"},
        "country": {"type": "text"}
      }
    },
     "dstname": {"type": "keyword"},
     "dst": {
       "properties": {
         "continent": {"type": "keyword"},
         "country": {"type": "keyword"}
       }
     },
     "origdst": {"type": "ip"},
     "origdstport": {"type": "integer"},
     "ipv": {"type": "integer"},
     "action": {"type": "keyword"},
     "msg": {"type": "text"},
     "class": {"type": "text"},
     "classification": {"type": "integer"},
     "alarm": {
       "properties": {
         "id": {"type": "integer"}
       } 
     },
     "target": {"type": "keyword"},
     "sensible": {"type": "integer"},
     "repeat": {"type": "integer"},
     "logtype": {"type": "keyword"},
     "service": {"type": "keyword"},
     "sent": {"type": "integer"},
     "rcvd": {"type": "integer"},
     "connection": {
       "properties": {
         "duration": {"type": "long"}
       }
     },
     "group": {
       "properties": {
         "id": {"type": "integer"}
       }
     },
     "op": {"type": "text"},
     "result": {"type": "long"},
     "arg": {"type": "text"},
     "protocol": {"type": "keyword"},
     "vulnid": {"type": "keyword"},
     "product": {"type": "text"},
     "detail": {"type": "text"},
     "family": {"type": "keyword"},
     "severity": {"type": "keyword"},
     "solution": {"type": "keyword"},
     "remote": {"type": "integer"},
     "targetclient": {"type": "integer"},
     "targetserver": {"type": "integer"},
     "discovery": {"type": "text"},
     "agentid": {"type": "integer"},
     "contentpolicy": {"type": "integer"},
     "mailruleid": {"type": "integer"},
     "cnruleid": {"type": "integer"},
     "remotenet": {"type": "ip"},
     "localnet": {"type": "ip"},
     "port": {"type": "integer"},
     "ikev": {"type": "integer"},
     "phase": {"type": "integer"},
     "side": {"type": "text"},
     "cookie_i": {"type": "text"},
     "filename": {"type": "text"},
     "virus": {"type": "text"},
     "urlrule":{
       "properties": {
         "id": {"type": "integer"}
       }
     },
     "cat_site": {"type": "text"},
     "method": {"type": "keyword"},
     "error": {"type": "integer"},
     "address": {"type": "ip"},
     "sessionid": {"type": "integer"}
    }
  } 
}

the Index:

PUT firewall-000001
{
  "aliases": {
    "firewall": {
      "is_write_index": true
    }
  }
}

Does Kibana see them as geopoints in index management?

I will confirm a little bit later, as It's the first time making a cluster with ELK, I had some problems so deleted all the index.

Once I restart my cluster I will give you more information about how the data is presented in Kibana

1 Like

Hello @warkolm again,

So I set up my cluster again and here is the information that you asked for,
So when I go to my index management, I did a filter of all the fields having the word geo and I am getting this list or result:

destination.geo.city_name
destination.geo.city_name.keyword
destination.geo.continent_code
destination.geo.continent_code.keyword
destination.geo.country_code2
destination.geo.country_code2.keyword
destination.geo.country_code3
destination.geo.country_code3.keyword
destination.geo.country_name
destination.geo.country_name.keyword
destination.geo.dma_code
destination.geo.ip
destination.geo.ip.keyword
destination.geo.latitude
destination.geo.location.lat
destination.geo.location.lon
destination.geo.longitude
destination.geo.postal_code
destination.geo.postal_code.keyword
destination.geo.region_code
destination.geo.region_code.keyword
destination.geo.region_name
destination.geo.region_name.keyword
destination.geo.timezone
destination.geo.timezone.keyword
source.geo.city_name
source.geo.city_name.keyword
source.geo.continent_code
source.geo.continent_code.keyword
source.geo.country_code2
source.geo.country_code2.keyword
source.geo.country_code3
source.geo.country_code3.keyword
source.geo.country_name
source.geo.country_name.keyword
source.geo.dma_code
source.geo.ip
source.geo.ip.keyword
source.geo.latitude
source.geo.location.lat
source.geo.location.lon
source.geo.longitude
source.geo.postal_code
source.geo.postal_code.keyword
source.geo.region_code
source.geo.region_code.keyword
source.geo.region_name
source.geo.region_name.keyword
source.geo.timezone
source.geo.timezone.keyword
source.nat.geo.city_name
source.nat.geo.city_name.keyword
source.nat.geo.continent_code
source.nat.geo.continent_code.keyword
source.nat.geo.country_code2
source.nat.geo.country_code2.keyword
source.nat.geo.country_code3
source.nat.geo.country_code3.keyword
source.nat.geo.country_name
source.nat.geo.country_name.keyword
source.nat.geo.dma_code
source.nat.geo.ip
source.nat.geo.ip.keyword
source.nat.geo.latitude
source.nat.geo.location.lat
source.nat.geo.location.lon
source.nat.geo.longitude
source.nat.geo.postal_code
source.nat.geo.postal_code.keyword
source.nat.geo.region_code
source.nat.geo.region_code.keyword
source.nat.geo.region_name
source.nat.geo.region_name.keyword
source.nat.geo.timezone
source.nat.geo.timezone.keyword

For more information about the type, searchable and Aggregation fields, see the bellow pictures:


Thanks

None of those are mapped as geopoints though.
Can you share your entire Logstash pipeline please.

Hello, so here is my global configuration:

input {
    udp {
      port => 5140
      type => "firewall"
    }
  }

filter {
  if [type] == "firewall" {

    grok {
      match => {
        "message" => [

      "\<%{NUMBER:Uknown:int}\>id=%{WORD:id} time=\"%{YEAR:year:int}-%{MONTHNUM:month:int}-%{MONTHDAY:day:int} %{TIME:time}\" fw=\"%{DATA:fw}\" tz=%{NUMBER:tz} startime=\"%{YEAR:year2:int}-%{MONTHNUM:month2:int}-%{MONTHDAY:day2:int} %{TIME:time2}\" (ipproto=\"%{WORD:network.protocol}\" )?(agentid=%{NUMBER:agentid:int} )?(pri=%{NUMBER:event.severity.code:int} )?(proto=%{USERNAME:proto} )?(confid=%{NUMBER:confid:int} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{USERNAME:rule.name}\" )?(op=%{WORD:op} )?(result=%{NUMBER:result} )?(user=\"%{DATA:user.name}\" )?(domain=\"%{DATA:domain}\" )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(confid=%{NUMBER:confid} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{DATA:rule.name}\" )?(mailruleid=%{NUMBER:mailruleid:int} )?(user=\"%{DATA:user.name}\" )?(cnruleid=%{NUMBER:cnruleid:int} )?(src=%{IP:source.ip:IP} )?(domain=\"%{URIPROTO:domain}\" )?(confid=%{NUMBER:confid:int} )?(ruleid=%{NUMBER:rule.id:int} )?(remotenet=%{IP:remotenet:IP} )?(localnet=%{IP:localnet:IP} )?(port=%{NUMBER:port:int} )?(srcport=%{NUMBER:source.port:int} )?(srcportname=%{USERNAME:source.port_name} )?(dst=%{IP:destination.ip:IP} )?(dstport=%{NUMBER:destination.port:int} )?(dstportname=%{WORD:destination.port_name} )?(srcname=%{USERNAME:srcname} )?(dst=%{IP:destination.ip:IP} )?(srcmac=%{MAC:source.mac} )?(dstname=%{USERNAME:dstname} )?(ikev=%{NUMBER:ikev:int} )?(phase=%{NUMBER:phase:int} )?(side=%{WORD:side} )?(cookie_i=%{DATA:cookie_i} )?(cookie_r=%{DATA:cookie_r} )?(localnet=%{IP:localnet}/%{NUMBER:localnet_masque} )?(remotenet=%{IP:remotenet}/%{NUMBER:remotenet_masque} )?(spi_in=%{DATA:spi_in} )?(spi_out=%{DATA:spi_out} )?(remoteid=%{IP:remoteid} )?(rulename=%{DATA:rule.name} )?(modsrc=%{IP:source.nat.ip:IP} )?(modsrcport=%{NUMBER:source.nat.port:int} )?(origdst=%{IP:origdst:IP} )?(origdstport=%{NUMBER:origdstport:int} )?(ipv=%{NUMBER:ipv:int} )?(sent=%{NUMBER:sent:int} )?(rcvd=%{NUMBER:rcvd:int} )?(duration=%{NUMBER:connection.duration:float} )?(filename=%{DATA:filename} )?(virus=%{WORD:virus} )?(srccontinent=\"%{WORD:src.continent}\" )?(srccountry=\"%{WORD:src.country}\" )?(srciprep=\"%{WORD:source.ip_reputation}\" )?(dstcontinent=\"%{WORD:dst.continent}\" )?(dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(action=%{WORD:action} )?(dstname=%{DATA:dstname} )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(urlruleid=%{NUMBER:urlrule.id:int} )?(cat_site=\"%{DATA:cat_site}\" )?(arg=\"%{DATA:arg}\" )?(method=\"%{DATA:method}\" )?(error=%{NUMBER:error:int} )?(user=\"%{DATA:user.name}\" )?(address=%{IP:address:IP} )?(sessionid=%{NUMBER:sessionid:int} )?(msg=\"%{DATA:msg}\" )?(dstcontinent=\"%{WORD:dst.continent}\" dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(duration=%{NUMBER:connection.duration:float} )?logtype=\"%{WORD:logtype}\""]
      }
    }

    if [event.severity.code] == 0 {
       mutate {
         add_field => {
           "event.severity.name" => "Emergency"
         }
       }
    } else if [event.severity.code] == 1 {
        mutate {
          add_field => {
            "event.severity.name" => "Alert"
           }
        }
    } else if [event.severity.code] == 2 {
       mutate {
         add_field => {
           "event.severity.name" => "Critical"
         }
       }
    } else if [event.severity.code] == 3 {
       mutate {
         add_field => {
           "event.severity.name" => "Error"
         }
       }
    } else if [event.severity.code] == 4 {
       mutate {
         add_field => {
           "event.severity.name" => "Warning"
         }
       }
    } else if [event.severity.code] == 5 {
       mutate {
         add_field => {
           "event.severity.name" => "Notice"
         }
       }
    } else if [event.severity.code] == 6 {
       mutate {
         add_field => {
           "event.severity.name" => "Information"
         }
       }
    } else if [event.severity.code] == 7 {
       mutate {
         add_field => {
           "event.severity.name" => "Debug"
         }
       }
    }

    mutate {
      add_field => {
        "timestamp" => "%{year2}-%{month2}-%{day2} %{time2}"
      }
    }

    date {
      match => [ "timestamp" , "YYYY-MM-dd HH:mm:ss"]
      target => "@timestamp"
    }

    mutate {
      remove_field => [ "timestamp", "year", "year2", "month", "month2", "day", "day2", "time", "time2" ] }

   geoip { source => "source.ip"
           target => "source.geo"
   }
   geoip { source => "destination.ip"
           target => "destination.geo"
    }
   geoip { source => "source.nat.ip"
           target => "source.nat.geo"
    }
  }

output {
  if [type] == "firewall" {
    elasticsearch {
      hosts => ["https://X.X.X.X:9200", "https://Y.Y.Y.Y:9200", "https://Z.Z.Z.Z:9200"]
      ilm_enabled => auto
      ilm_rollover_alias => "firewall"
      ilm_pattern => "000001"
      ilm_policy => "hot-warm-cold-delete-6months-policy"
      template_name => "firewall"
      cacert => '/etc/logstash/certs/ca.crt'
      user => "elastic"
      password => "password"
    }
  }
}

So after some days of searching, I have added this to my template :

     "source": {
       "properties": {
          "geo": {
            "properties": {
              "continent_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "region_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "city_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "location": {
                "type": "geo_point"
              },
              "region_name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },

and when I check the template now I am seeing this:

 "source": {
        "properties": {
          "geo": {
            "properties": {
              "continent_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "region_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "city_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "location": {
                "type": "geo_point"
              },
              "region_name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },

so it took on consideration what I added, but in the index mapping, I am not seeing any geo_point

        "source": {
          "properties": {
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_code2": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_code3": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "dma_code": {
                  "type": "long"
                },
                "ip": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "latitude": {
                  "type": "float"
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "longitude": {
                  "type": "float"
                },
                "postal_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "timezone": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }

Could somone please help to find out what I am missing, ? (I already refreshed the index, but still not seeing geo_point type)