Create point to point map from Logstash pipeline

Elastic Stack Kibana
1

Hello,

I am using logstash to parse my firewall logs and enrich them with geo location information

    geoip { source => "source.ip"
            target => "source.geo"
    }
    geoip { source => "destination.ip"
            target => "destination.geo"
     }
    geoip { source => "source.nat.ip"
            target => "source.nat.geo"
     }
   }

I want to create a point to point map using these informations, but when I click on the map I can't see my firewall-* index which contain my firewall logs

image

Could you please help me to understand why this is happening ?

Thanks

2

Didn't see the output index in the config.. Is there logstash output index pointed to firewall-*?
If yes, just create the index pattern based on the name firewall-*

3

I already have the index named firewall-*, the only problem is that it doesn't appear in the point to point map.

NB: The firewall-* index appear in the EMS Boundaries map

image

4

Hello, I've the same exact problem. Did you have any news on this topic?
Thanks Gianluca

5

Are the geopoints mapped correctly in the index?

6

@gian72 didn't find a solution yet

@warkolm, I am creating the index and the mapping like that:

the template:

PUT _template/firewall
{
  "index_patterns": ["firewall-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "hot-warm-cold-delete-6months-policy",  
    "index.lifecycle.rollover_alias": "firewall"
  },
  "mappings": {
    "properties": {
     "event": {
       "properties": {
         "severity": {
           "properties": {
             "code": { "type": "integer"},
             "name": { "type": "keyword"}
           }
         }
       }
     },
     "confid": {"type": "integer" },
     "slotlevel": {"type": "integer"},
     "rule": {
       "properties": {
         "id": { "type": "integer"},
         "name": {"type": "keyword" }
       }
     },
     "user": {
       "properties": {
         "name": {"type": "keyword"}
       }
     },
     "domain": {"type": "text"},
     "source": {
       "properties": {
         "interface_name": {"type": "keyword"},
         "ip":{ "type": "ip"},
         "ip_reputation": {"type": "keyword"},
         "port": {"type": "integer"},
         "port_name": {"type": "keyword" },
         "mac": {"type": "keyword"},
         "nat": {
           "properties":{
             "ip": {"type": "ip"},
             "port": {"type": "integer"}
           }
         }
       }
     },
     "srcifname": {"type": "keyword"},
     "network": {
       "properties": {
         "protocol": { "type": "keyword"}
       }
     },
     "destination": {
       "properties": {
         "interface_name": {"type": "keyword"},
         "ip": { "type": "ip"},
         "ip_reputation": {"type": "keyword"},
         "port": {"type": "integer"},
         "port_name": {"type": "keyword"}
       }
     },
     "dstifname": {"type": "text"},
     "icmp": {
       "properties": {
         "type": { "type": "integer"},
         "code": { "type": "integer"}
       }
     },
     "proto": {"type": "keyword"},
     "srcname": {"type": "keyword"},
    "src": {
      "properties": {
        "continent": {"type": "keyword"},
        "country": {"type": "text"}
      }
    },
     "dstname": {"type": "keyword"},
     "dst": {
       "properties": {
         "continent": {"type": "keyword"},
         "country": {"type": "keyword"}
       }
     },
     "origdst": {"type": "ip"},
     "origdstport": {"type": "integer"},
     "ipv": {"type": "integer"},
     "action": {"type": "keyword"},
     "msg": {"type": "text"},
     "class": {"type": "text"},
     "classification": {"type": "integer"},
     "alarm": {
       "properties": {
         "id": {"type": "integer"}
       } 
     },
     "target": {"type": "keyword"},
     "sensible": {"type": "integer"},
     "repeat": {"type": "integer"},
     "logtype": {"type": "keyword"},
     "service": {"type": "keyword"},
     "sent": {"type": "integer"},
     "rcvd": {"type": "integer"},
     "connection": {
       "properties": {
         "duration": {"type": "long"}
       }
     },
     "group": {
       "properties": {
         "id": {"type": "integer"}
       }
     },
     "op": {"type": "text"},
     "result": {"type": "long"},
     "arg": {"type": "text"},
     "protocol": {"type": "keyword"},
     "vulnid": {"type": "keyword"},
     "product": {"type": "text"},
     "detail": {"type": "text"},
     "family": {"type": "keyword"},
     "severity": {"type": "keyword"},
     "solution": {"type": "keyword"},
     "remote": {"type": "integer"},
     "targetclient": {"type": "integer"},
     "targetserver": {"type": "integer"},
     "discovery": {"type": "text"},
     "agentid": {"type": "integer"},
     "contentpolicy": {"type": "integer"},
     "mailruleid": {"type": "integer"},
     "cnruleid": {"type": "integer"},
     "remotenet": {"type": "ip"},
     "localnet": {"type": "ip"},
     "port": {"type": "integer"},
     "ikev": {"type": "integer"},
     "phase": {"type": "integer"},
     "side": {"type": "text"},
     "cookie_i": {"type": "text"},
     "filename": {"type": "text"},
     "virus": {"type": "text"},
     "urlrule":{
       "properties": {
         "id": {"type": "integer"}
       }
     },
     "cat_site": {"type": "text"},
     "method": {"type": "keyword"},
     "error": {"type": "integer"},
     "address": {"type": "ip"},
     "sessionid": {"type": "integer"}
    }
  } 
}

the Index:

PUT firewall-000001
{
  "aliases": {
    "firewall": {
      "is_write_index": true
    }
  }
}
7

Does Kibana see them as geopoints in index management?

8

I will confirm a little bit later, as It's the first time making a cluster with ELK, I had some problems so deleted all the index.

Once I restart my cluster I will give you more information about how the data is presented in Kibana

1 Like
9

Hello @warkolm again,

So I set up my cluster again and here is the information that you asked for,
So when I go to my index management, I did a filter of all the fields having the word geo and I am getting this list or result:

destination.geo.city_name
destination.geo.city_name.keyword
destination.geo.continent_code
destination.geo.continent_code.keyword
destination.geo.country_code2
destination.geo.country_code2.keyword
destination.geo.country_code3
destination.geo.country_code3.keyword
destination.geo.country_name
destination.geo.country_name.keyword
destination.geo.dma_code
destination.geo.ip
destination.geo.ip.keyword
destination.geo.latitude
destination.geo.location.lat
destination.geo.location.lon
destination.geo.longitude
destination.geo.postal_code
destination.geo.postal_code.keyword
destination.geo.region_code
destination.geo.region_code.keyword
destination.geo.region_name
destination.geo.region_name.keyword
destination.geo.timezone
destination.geo.timezone.keyword
source.geo.city_name
source.geo.city_name.keyword
source.geo.continent_code
source.geo.continent_code.keyword
source.geo.country_code2
source.geo.country_code2.keyword
source.geo.country_code3
source.geo.country_code3.keyword
source.geo.country_name
source.geo.country_name.keyword
source.geo.dma_code
source.geo.ip
source.geo.ip.keyword
source.geo.latitude
source.geo.location.lat
source.geo.location.lon
source.geo.longitude
source.geo.postal_code
source.geo.postal_code.keyword
source.geo.region_code
source.geo.region_code.keyword
source.geo.region_name
source.geo.region_name.keyword
source.geo.timezone
source.geo.timezone.keyword
source.nat.geo.city_name
source.nat.geo.city_name.keyword
source.nat.geo.continent_code
source.nat.geo.continent_code.keyword
source.nat.geo.country_code2
source.nat.geo.country_code2.keyword
source.nat.geo.country_code3
source.nat.geo.country_code3.keyword
source.nat.geo.country_name
source.nat.geo.country_name.keyword
source.nat.geo.dma_code
source.nat.geo.ip
source.nat.geo.ip.keyword
source.nat.geo.latitude
source.nat.geo.location.lat
source.nat.geo.location.lon
source.nat.geo.longitude
source.nat.geo.postal_code
source.nat.geo.postal_code.keyword
source.nat.geo.region_code
source.nat.geo.region_code.keyword
source.nat.geo.region_name
source.nat.geo.region_name.keyword
source.nat.geo.timezone
source.nat.geo.timezone.keyword

For more information about the type, searchable and Aggregation fields, see the bellow pictures:

image
image1169×556 51.5 KB

image
image1160×558 46.3 KB

image
image1156×546 47.7 KB

image
image1157×513 42.6 KB

image
image1171×483 43.2 KB

image
image1158×560 50.1 KB

image
image1146×553 47.7 KB

image
image1163×305 25.7 KB

Thanks

10

None of those are mapped as geopoints though.
Can you share your entire Logstash pipeline please.

11

Hello, so here is my global configuration:

input {
    udp {
      port => 5140
      type => "firewall"
    }
  }

filter {
  if [type] == "firewall" {

    grok {
      match => {
        "message" => [

      "\<%{NUMBER:Uknown:int}\>id=%{WORD:id} time=\"%{YEAR:year:int}-%{MONTHNUM:month:int}-%{MONTHDAY:day:int} %{TIME:time}\" fw=\"%{DATA:fw}\" tz=%{NUMBER:tz} startime=\"%{YEAR:year2:int}-%{MONTHNUM:month2:int}-%{MONTHDAY:day2:int} %{TIME:time2}\" (ipproto=\"%{WORD:network.protocol}\" )?(agentid=%{NUMBER:agentid:int} )?(pri=%{NUMBER:event.severity.code:int} )?(proto=%{USERNAME:proto} )?(confid=%{NUMBER:confid:int} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{USERNAME:rule.name}\" )?(op=%{WORD:op} )?(result=%{NUMBER:result} )?(user=\"%{DATA:user.name}\" )?(domain=\"%{DATA:domain}\" )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(confid=%{NUMBER:confid} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{DATA:rule.name}\" )?(mailruleid=%{NUMBER:mailruleid:int} )?(user=\"%{DATA:user.name}\" )?(cnruleid=%{NUMBER:cnruleid:int} )?(src=%{IP:source.ip:IP} )?(domain=\"%{URIPROTO:domain}\" )?(confid=%{NUMBER:confid:int} )?(ruleid=%{NUMBER:rule.id:int} )?(remotenet=%{IP:remotenet:IP} )?(localnet=%{IP:localnet:IP} )?(port=%{NUMBER:port:int} )?(srcport=%{NUMBER:source.port:int} )?(srcportname=%{USERNAME:source.port_name} )?(dst=%{IP:destination.ip:IP} )?(dstport=%{NUMBER:destination.port:int} )?(dstportname=%{WORD:destination.port_name} )?(srcname=%{USERNAME:srcname} )?(dst=%{IP:destination.ip:IP} )?(srcmac=%{MAC:source.mac} )?(dstname=%{USERNAME:dstname} )?(ikev=%{NUMBER:ikev:int} )?(phase=%{NUMBER:phase:int} )?(side=%{WORD:side} )?(cookie_i=%{DATA:cookie_i} )?(cookie_r=%{DATA:cookie_r} )?(localnet=%{IP:localnet}/%{NUMBER:localnet_masque} )?(remotenet=%{IP:remotenet}/%{NUMBER:remotenet_masque} )?(spi_in=%{DATA:spi_in} )?(spi_out=%{DATA:spi_out} )?(remoteid=%{IP:remoteid} )?(rulename=%{DATA:rule.name} )?(modsrc=%{IP:source.nat.ip:IP} )?(modsrcport=%{NUMBER:source.nat.port:int} )?(origdst=%{IP:origdst:IP} )?(origdstport=%{NUMBER:origdstport:int} )?(ipv=%{NUMBER:ipv:int} )?(sent=%{NUMBER:sent:int} )?(rcvd=%{NUMBER:rcvd:int} )?(duration=%{NUMBER:connection.duration:float} )?(filename=%{DATA:filename} )?(virus=%{WORD:virus} )?(srccontinent=\"%{WORD:src.continent}\" )?(srccountry=\"%{WORD:src.country}\" )?(srciprep=\"%{WORD:source.ip_reputation}\" )?(dstcontinent=\"%{WORD:dst.continent}\" )?(dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(action=%{WORD:action} )?(dstname=%{DATA:dstname} )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(urlruleid=%{NUMBER:urlrule.id:int} )?(cat_site=\"%{DATA:cat_site}\" )?(arg=\"%{DATA:arg}\" )?(method=\"%{DATA:method}\" )?(error=%{NUMBER:error:int} )?(user=\"%{DATA:user.name}\" )?(address=%{IP:address:IP} )?(sessionid=%{NUMBER:sessionid:int} )?(msg=\"%{DATA:msg}\" )?(dstcontinent=\"%{WORD:dst.continent}\" dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(duration=%{NUMBER:connection.duration:float} )?logtype=\"%{WORD:logtype}\""]
      }
    }

    if [event.severity.code] == 0 {
       mutate {
         add_field => {
           "event.severity.name" => "Emergency"
         }
       }
    } else if [event.severity.code] == 1 {
        mutate {
          add_field => {
            "event.severity.name" => "Alert"
           }
        }
    } else if [event.severity.code] == 2 {
       mutate {
         add_field => {
           "event.severity.name" => "Critical"
         }
       }
    } else if [event.severity.code] == 3 {
       mutate {
         add_field => {
           "event.severity.name" => "Error"
         }
       }
    } else if [event.severity.code] == 4 {
       mutate {
         add_field => {
           "event.severity.name" => "Warning"
         }
       }
    } else if [event.severity.code] == 5 {
       mutate {
         add_field => {
           "event.severity.name" => "Notice"
         }
       }
    } else if [event.severity.code] == 6 {
       mutate {
         add_field => {
           "event.severity.name" => "Information"
         }
       }
    } else if [event.severity.code] == 7 {
       mutate {
         add_field => {
           "event.severity.name" => "Debug"
         }
       }
    }

    mutate {
      add_field => {
        "timestamp" => "%{year2}-%{month2}-%{day2} %{time2}"
      }
    }

    date {
      match => [ "timestamp" , "YYYY-MM-dd HH:mm:ss"]
      target => "@timestamp"
    }

    mutate {
      remove_field => [ "timestamp", "year", "year2", "month", "month2", "day", "day2", "time", "time2" ] }

   geoip { source => "source.ip"
           target => "source.geo"
   }
   geoip { source => "destination.ip"
           target => "destination.geo"
    }
   geoip { source => "source.nat.ip"
           target => "source.nat.geo"
    }
  }

output {
  if [type] == "firewall" {
    elasticsearch {
      hosts => ["https://X.X.X.X:9200", "https://Y.Y.Y.Y:9200", "https://Z.Z.Z.Z:9200"]
      ilm_enabled => auto
      ilm_rollover_alias => "firewall"
      ilm_pattern => "000001"
      ilm_policy => "hot-warm-cold-delete-6months-policy"
      template_name => "firewall"
      cacert => '/etc/logstash/certs/ca.crt'
      user => "elastic"
      password => "password"
    }
  }
}
12

So after some days of searching, I have added this to my template :

     "source": {
       "properties": {
          "geo": {
            "properties": {
              "continent_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "region_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "city_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "location": {
                "type": "geo_point"
              },
              "region_name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },

and when I check the template now I am seeing this:

 "source": {
        "properties": {
          "geo": {
            "properties": {
              "continent_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "region_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "city_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_iso_code": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "country_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "location": {
                "type": "geo_point"
              },
              "region_name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },

so it took on consideration what I added, but in the index mapping, I am not seeing any geo_point

        "source": {
          "properties": {
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_code2": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_code3": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "dma_code": {
                  "type": "long"
                },
                "ip": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "latitude": {
                  "type": "float"
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "longitude": {
                  "type": "float"
                },
                "postal_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "timezone": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }

image
image1199×304 24.1 KB

Could somone please help to find out what I am missing, ? (I already refreshed the index, but still not seeing geo_point type)

13

Any help please !

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.