# Create point to point map from Logstash pipeline
**URL:** https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722
**Category:** Kibana
**Created:** [December 28, 2020, 10:58am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722 "2020-12-28T10:58:35Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [December 28, 2020, 10:58am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/1 "2020-12-28T10:58:35Z")
Hello,
I am using logstash to parse my firewall logs and enrich them with geo location information
```auto
geoip { source => "source.ip"
target => "source.geo"
}
geoip { source => "destination.ip"
target => "destination.geo"
}
geoip { source => "source.nat.ip"
target => "source.nat.geo"
}
}
```
I want to create a point to point map using these informations, but when I click on the map I can't see my `firewall-*` index which contain my firewall logs
Could you please help me to understand why this is happening ?
Thanks
---
### Author:  [@fadjar340](https://discuss.elastic.co/u/fadjar340)
#### Post date: [December 28, 2020, 11:40am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/2 "2020-12-28T11:40:50Z")
Didn't see the output index in the config.. Is there logstash output index pointed to `firewall-*`?
If yes, just create the index pattern based on the name `firewall-*`
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [December 28, 2020, 1:00pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/3 "2020-12-28T13:00:41Z")
I already have the index named `firewall-*`, the only problem is that it doesn't appear in the point to point map.
**NB:** The `firewall-*` index appear in the `EMS Boundaries` map
---
### Author:  [@gian72](https://discuss.elastic.co/u/gian72)
#### Post date: [December 30, 2020, 6:13pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/4 "2020-12-30T18:13:26Z")
Hello, I've the same exact problem. Did you have any news on this topic?
Thanks Gianluca
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 30, 2020, 10:15pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/5 "2020-12-30T22:15:04Z")
Are the geopoints mapped correctly in the index?
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [December 31, 2020, 8:05am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/6 "2020-12-31T08:05:02Z")
@gian72 didn't find a solution yet
@warkolm, I am creating the index and the mapping like that:
**the template:**
```auto
PUT _template/firewall
{
"index_patterns": ["firewall-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "hot-warm-cold-delete-6months-policy",
"index.lifecycle.rollover_alias": "firewall"
},
"mappings": {
"properties": {
"event": {
"properties": {
"severity": {
"properties": {
"code": { "type": "integer"},
"name": { "type": "keyword"}
}
}
}
},
"confid": {"type": "integer" },
"slotlevel": {"type": "integer"},
"rule": {
"properties": {
"id": { "type": "integer"},
"name": {"type": "keyword" }
}
},
"user": {
"properties": {
"name": {"type": "keyword"}
}
},
"domain": {"type": "text"},
"source": {
"properties": {
"interface_name": {"type": "keyword"},
"ip":{ "type": "ip"},
"ip_reputation": {"type": "keyword"},
"port": {"type": "integer"},
"port_name": {"type": "keyword" },
"mac": {"type": "keyword"},
"nat": {
"properties":{
"ip": {"type": "ip"},
"port": {"type": "integer"}
}
}
}
},
"srcifname": {"type": "keyword"},
"network": {
"properties": {
"protocol": { "type": "keyword"}
}
},
"destination": {
"properties": {
"interface_name": {"type": "keyword"},
"ip": { "type": "ip"},
"ip_reputation": {"type": "keyword"},
"port": {"type": "integer"},
"port_name": {"type": "keyword"}
}
},
"dstifname": {"type": "text"},
"icmp": {
"properties": {
"type": { "type": "integer"},
"code": { "type": "integer"}
}
},
"proto": {"type": "keyword"},
"srcname": {"type": "keyword"},
"src": {
"properties": {
"continent": {"type": "keyword"},
"country": {"type": "text"}
}
},
"dstname": {"type": "keyword"},
"dst": {
"properties": {
"continent": {"type": "keyword"},
"country": {"type": "keyword"}
}
},
"origdst": {"type": "ip"},
"origdstport": {"type": "integer"},
"ipv": {"type": "integer"},
"action": {"type": "keyword"},
"msg": {"type": "text"},
"class": {"type": "text"},
"classification": {"type": "integer"},
"alarm": {
"properties": {
"id": {"type": "integer"}
}
},
"target": {"type": "keyword"},
"sensible": {"type": "integer"},
"repeat": {"type": "integer"},
"logtype": {"type": "keyword"},
"service": {"type": "keyword"},
"sent": {"type": "integer"},
"rcvd": {"type": "integer"},
"connection": {
"properties": {
"duration": {"type": "long"}
}
},
"group": {
"properties": {
"id": {"type": "integer"}
}
},
"op": {"type": "text"},
"result": {"type": "long"},
"arg": {"type": "text"},
"protocol": {"type": "keyword"},
"vulnid": {"type": "keyword"},
"product": {"type": "text"},
"detail": {"type": "text"},
"family": {"type": "keyword"},
"severity": {"type": "keyword"},
"solution": {"type": "keyword"},
"remote": {"type": "integer"},
"targetclient": {"type": "integer"},
"targetserver": {"type": "integer"},
"discovery": {"type": "text"},
"agentid": {"type": "integer"},
"contentpolicy": {"type": "integer"},
"mailruleid": {"type": "integer"},
"cnruleid": {"type": "integer"},
"remotenet": {"type": "ip"},
"localnet": {"type": "ip"},
"port": {"type": "integer"},
"ikev": {"type": "integer"},
"phase": {"type": "integer"},
"side": {"type": "text"},
"cookie_i": {"type": "text"},
"filename": {"type": "text"},
"virus": {"type": "text"},
"urlrule":{
"properties": {
"id": {"type": "integer"}
}
},
"cat_site": {"type": "text"},
"method": {"type": "keyword"},
"error": {"type": "integer"},
"address": {"type": "ip"},
"sessionid": {"type": "integer"}
}
}
}
```
**the Index:**
```auto
PUT firewall-000001
{
"aliases": {
"firewall": {
"is_write_index": true
}
}
}
```
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 31, 2020, 8:20am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/7 "2020-12-31T08:20:20Z")
Does Kibana see them as geopoints in index management?
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [December 31, 2020, 8:26am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/8 "2020-12-31T08:26:05Z")
I will confirm a little bit later, as It's the first time making a cluster with ELK, I had some problems so deleted all the index.
Once I restart my cluster I will give you more information about how the data is presented in Kibana
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 4, 2021, 4:47pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/9 "2021-01-04T16:47:20Z")
Hello @warkolm again,
So I set up my cluster again and here is the information that you asked for,
So when I go to my index management, I did a filter of all the fields having the word `geo` and I am getting this list or result:
```auto
destination.geo.city_name
destination.geo.city_name.keyword
destination.geo.continent_code
destination.geo.continent_code.keyword
destination.geo.country_code2
destination.geo.country_code2.keyword
destination.geo.country_code3
destination.geo.country_code3.keyword
destination.geo.country_name
destination.geo.country_name.keyword
destination.geo.dma_code
destination.geo.ip
destination.geo.ip.keyword
destination.geo.latitude
destination.geo.location.lat
destination.geo.location.lon
destination.geo.longitude
destination.geo.postal_code
destination.geo.postal_code.keyword
destination.geo.region_code
destination.geo.region_code.keyword
destination.geo.region_name
destination.geo.region_name.keyword
destination.geo.timezone
destination.geo.timezone.keyword
source.geo.city_name
source.geo.city_name.keyword
source.geo.continent_code
source.geo.continent_code.keyword
source.geo.country_code2
source.geo.country_code2.keyword
source.geo.country_code3
source.geo.country_code3.keyword
source.geo.country_name
source.geo.country_name.keyword
source.geo.dma_code
source.geo.ip
source.geo.ip.keyword
source.geo.latitude
source.geo.location.lat
source.geo.location.lon
source.geo.longitude
source.geo.postal_code
source.geo.postal_code.keyword
source.geo.region_code
source.geo.region_code.keyword
source.geo.region_name
source.geo.region_name.keyword
source.geo.timezone
source.geo.timezone.keyword
source.nat.geo.city_name
source.nat.geo.city_name.keyword
source.nat.geo.continent_code
source.nat.geo.continent_code.keyword
source.nat.geo.country_code2
source.nat.geo.country_code2.keyword
source.nat.geo.country_code3
source.nat.geo.country_code3.keyword
source.nat.geo.country_name
source.nat.geo.country_name.keyword
source.nat.geo.dma_code
source.nat.geo.ip
source.nat.geo.ip.keyword
source.nat.geo.latitude
source.nat.geo.location.lat
source.nat.geo.location.lon
source.nat.geo.longitude
source.nat.geo.postal_code
source.nat.geo.postal_code.keyword
source.nat.geo.region_code
source.nat.geo.region_code.keyword
source.nat.geo.region_name
source.nat.geo.region_name.keyword
source.nat.geo.timezone
source.nat.geo.timezone.keyword
```
For more information about the `type`, `searchable` and `Aggregation` fields, see the bellow pictures:
Thanks
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 4, 2021, 10:00pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/10 "2021-01-04T22:00:28Z")
None of those are mapped as geopoints though.
Can you share your entire Logstash pipeline please.
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 5, 2021, 8:23am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/11 "2021-01-05T08:23:57Z")
Hello, so here is my global configuration:
```auto
input {
udp {
port => 5140
type => "firewall"
}
}
filter {
if [type] == "firewall" {
grok {
match => {
"message" => [
"\<%{NUMBER:Uknown:int}\>id=%{WORD:id} time=\"%{YEAR:year:int}-%{MONTHNUM:month:int}-%{MONTHDAY:day:int} %{TIME:time}\" fw=\"%{DATA:fw}\" tz=%{NUMBER:tz} startime=\"%{YEAR:year2:int}-%{MONTHNUM:month2:int}-%{MONTHDAY:day2:int} %{TIME:time2}\" (ipproto=\"%{WORD:network.protocol}\" )?(agentid=%{NUMBER:agentid:int} )?(pri=%{NUMBER:event.severity.code:int} )?(proto=%{USERNAME:proto} )?(confid=%{NUMBER:confid:int} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{USERNAME:rule.name}\" )?(op=%{WORD:op} )?(result=%{NUMBER:result} )?(user=\"%{DATA:user.name}\" )?(domain=\"%{DATA:domain}\" )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(confid=%{NUMBER:confid} )?(slotlevel=%{NUMBER:slotlevel:int} )?(ruleid=%{NUMBER:rule.id:int} )?(rulename=\"%{DATA:rule.name}\" )?(mailruleid=%{NUMBER:mailruleid:int} )?(user=\"%{DATA:user.name}\" )?(cnruleid=%{NUMBER:cnruleid:int} )?(src=%{IP:source.ip:IP} )?(domain=\"%{URIPROTO:domain}\" )?(confid=%{NUMBER:confid:int} )?(ruleid=%{NUMBER:rule.id:int} )?(remotenet=%{IP:remotenet:IP} )?(localnet=%{IP:localnet:IP} )?(port=%{NUMBER:port:int} )?(srcport=%{NUMBER:source.port:int} )?(srcportname=%{USERNAME:source.port_name} )?(dst=%{IP:destination.ip:IP} )?(dstport=%{NUMBER:destination.port:int} )?(dstportname=%{WORD:destination.port_name} )?(srcname=%{USERNAME:srcname} )?(dst=%{IP:destination.ip:IP} )?(srcmac=%{MAC:source.mac} )?(dstname=%{USERNAME:dstname} )?(ikev=%{NUMBER:ikev:int} )?(phase=%{NUMBER:phase:int} )?(side=%{WORD:side} )?(cookie_i=%{DATA:cookie_i} )?(cookie_r=%{DATA:cookie_r} )?(localnet=%{IP:localnet}/%{NUMBER:localnet_masque} )?(remotenet=%{IP:remotenet}/%{NUMBER:remotenet_masque} )?(spi_in=%{DATA:spi_in} )?(spi_out=%{DATA:spi_out} )?(remoteid=%{IP:remoteid} )?(rulename=%{DATA:rule.name} )?(modsrc=%{IP:source.nat.ip:IP} )?(modsrcport=%{NUMBER:source.nat.port:int} )?(origdst=%{IP:origdst:IP} )?(origdstport=%{NUMBER:origdstport:int} )?(ipv=%{NUMBER:ipv:int} )?(sent=%{NUMBER:sent:int} )?(rcvd=%{NUMBER:rcvd:int} )?(duration=%{NUMBER:connection.duration:float} )?(filename=%{DATA:filename} )?(virus=%{WORD:virus} )?(srccontinent=\"%{WORD:src.continent}\" )?(srccountry=\"%{WORD:src.country}\" )?(srciprep=\"%{WORD:source.ip_reputation}\" )?(dstcontinent=\"%{WORD:dst.continent}\" )?(dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(action=%{WORD:action} )?(dstname=%{DATA:dstname} )?(contentpolicy=%{NUMBER:contentpolicy:int} )?(urlruleid=%{NUMBER:urlrule.id:int} )?(cat_site=\"%{DATA:cat_site}\" )?(arg=\"%{DATA:arg}\" )?(method=\"%{DATA:method}\" )?(error=%{NUMBER:error:int} )?(user=\"%{DATA:user.name}\" )?(address=%{IP:address:IP} )?(sessionid=%{NUMBER:sessionid:int} )?(msg=\"%{DATA:msg}\" )?(dstcontinent=\"%{WORD:dst.continent}\" dstcountry=\"%{WORD:dst.country}\" )?(dstiprep=\"%{DATA:destination.ip_reputation}\" )?(duration=%{NUMBER:connection.duration:float} )?logtype=\"%{WORD:logtype}\""]
}
}
if [event.severity.code] == 0 {
mutate {
add_field => {
"event.severity.name" => "Emergency"
}
}
} else if [event.severity.code] == 1 {
mutate {
add_field => {
"event.severity.name" => "Alert"
}
}
} else if [event.severity.code] == 2 {
mutate {
add_field => {
"event.severity.name" => "Critical"
}
}
} else if [event.severity.code] == 3 {
mutate {
add_field => {
"event.severity.name" => "Error"
}
}
} else if [event.severity.code] == 4 {
mutate {
add_field => {
"event.severity.name" => "Warning"
}
}
} else if [event.severity.code] == 5 {
mutate {
add_field => {
"event.severity.name" => "Notice"
}
}
} else if [event.severity.code] == 6 {
mutate {
add_field => {
"event.severity.name" => "Information"
}
}
} else if [event.severity.code] == 7 {
mutate {
add_field => {
"event.severity.name" => "Debug"
}
}
}
mutate {
add_field => {
"timestamp" => "%{year2}-%{month2}-%{day2} %{time2}"
}
}
date {
match => ["timestamp" , "YYYY-MM-dd HH:mm:ss"]
target => "@timestamp"
}
mutate {
remove_field => ["timestamp", "year", "year2", "month", "month2", "day", "day2", "time", "time2"] }
geoip { source => "source.ip"
target => "source.geo"
}
geoip { source => "destination.ip"
target => "destination.geo"
}
geoip { source => "source.nat.ip"
target => "source.nat.geo"
}
}
output {
if [type] == "firewall" {
elasticsearch {
hosts => ["https://X.X.X.X:9200", "https://Y.Y.Y.Y:9200", "https://Z.Z.Z.Z:9200"]
ilm_enabled => auto
ilm_rollover_alias => "firewall"
ilm_pattern => "000001"
ilm_policy => "hot-warm-cold-delete-6months-policy"
template_name => "firewall"
cacert => '/etc/logstash/certs/ca.crt'
user => "elastic"
password => "password"
}
}
}
```
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 14, 2021, 9:51am UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/12 "2021-01-14T09:51:23Z")
So after some days of searching, I have added this to my template :
```auto
"source": {
"properties": {
"geo": {
"properties": {
"continent_name": {
"ignore_above": 1024,
"type": "keyword"
},
"region_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"city_name": {
"ignore_above": 1024,
"type": "keyword"
},
"country_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"country_name": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"location": {
"type": "geo_point"
},
"region_name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
```
and when I check the template now I am seeing this:
```auto
"source": {
"properties": {
"geo": {
"properties": {
"continent_name": {
"ignore_above": 1024,
"type": "keyword"
},
"region_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"city_name": {
"ignore_above": 1024,
"type": "keyword"
},
"country_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"country_name": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"location": {
"type": "geo_point"
},
"region_name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
```
so it took on consideration what I added, but in the index mapping, I am not seeing any `geo_point`
```auto
"source": {
"properties": {
"geo": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
```
Could somone please help to find out what I am missing, ? (I already refreshed the index, but still not seeing `geo_point` type)
---
### Author:  [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)
#### Post date: [January 27, 2021, 12:59pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/13 "2021-01-27T12:59:44Z")
Any help please !
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 24, 2021, 12:59pm UTC](https://discuss.elastic.co/t/create-point-to-point-map-from-logstash-pipeline/259722/14 "2021-02-24T12:59:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.