Can we create two GeoIP Filters in one logstash config file?

Elastic Stack Logstash
1

Hi Experts,

My requirement is to create 2 maps , one is for Source IP and other is for Destination IP.

For Source IP what I have done is I used GeoIP filter as below
geoip { source => "src"}

Now I am trying the same for Destination geoip { source => "dst"}, but in the map visualization I can only see geoip.location, now confusion is how I can select for src or dst.

Thanks
VG

2

So I have done this but still in Kibana I do not see two fileds.

filter {
geoip {
source => "src"
target => "src_geoip"
}
geoip {
source => "dst"
target => "dst_geoip"
}
}

3

That's because those fields aren't mapped as geo_point. You need to adjust the Logstash index template. See the related options in the documentation of the elasticsearch output.

4

@magnusbaeck,

Thanks for the reply , so I have following index template.
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}

Do I need to make any changes to it ?

5

Yes. That mapping is for fields named geoip. Your fields are named src_geoip and dst_geoip.

6

@magnusbaeck Ah!!! Now I get what you are saying , it works for me . Thanks alot, you are always helpful .

7

Good Afternoon,

Forgive me for bringing an old thread back to life but I am still struggling with how to get this to work for both dst and src ip addresses for a geoip lookup. I have

filter {
geoip {
source => "src"
target => "src_geoip"
}
geoip {
source => "dst"
target => "dst_geoip"
}
}

and the corresponding fields come into the index inside of elasticsearch just fine. The src_geoip.location and dst_geoip.location fields look to be in the correct format for geojson. Can you assist me in what commands I need to enter to change the type of these two fields in my mapping to geo points and not numbers as they stand today.
We are currently just using the default index that is generated from the elasticsearch output from logstash. My geopoint field looks the same as above.

Please get back to me at your earliest convenience. Thank you.

Cody Betsworth

8

If you need me to provide any additional information just ask and I can provide whatever you need to assist. Been struggling with this one for awhile. Thank you in advance to anyone willing to help.

Cody

9

You need to update the index template used for your indexes to map the src_geoip and dst_geoip fields as geo_point. By default Logstash's elasticsearch output uploads the index template to use and there are a few configuration options that control the exact behavior. Make a copy of the current index template and make the necessary adjustments there.

For further help please ask a more specific question. At least I don't have time to write a full step-by-step for this, but I'm sure it's been before by other people (here, in the documentation, in blog posts and/or on StackOverflow).

11

Hi Cody,

This is what you need to do . Assuming you are using ES1.7.X, because things are slight different if you are on ES2.X

step 1 --> In LS you need to update target for src_geoip and dst_geoip as geoip , something like (be sure src and dst fields should be IP type)

geoip { source => "src" target => "srcgeoip" }
geoip {source => "dst" target => "dstgeoip" }

Step 2) In ES template or using API you need to map fields to Geo_point

"srcgeoip" : {"type" : "object","dynamic": true,"path": "full","properties" : {"location" : { "type" : "geo_point" }}},
"dstgeoip" : {"type" : "object","dynamic": true,"path": "full","properties" : {"location" : { "type" : "geo_point" }}}

Step 3) now parse your data and you will see something like this in Kibana4.1.1

Let me know if you have more queries or concern on this

Thanks
VG

12

I am currently using elasticsearch 2.3.

{
"name" : "elk02",
"cluster_name" : "ArtOfSteal2",
"version" : {
"number" : "2.3.3",
"build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"build_timestamp" : "2016-05-17T15:40:04Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}

Here is my elasticsearch-fortinet.json template I am trying to use. I tried to just simply add your recommendations below the geoip field below but it still does not seem to be working. Thank you again for all your help. You mentioned 2.x elasticsearch is a little different on how I need to implement this request. Get back to me at your earliest convenience.

{
"template" : "fortinet*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date" },
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
"src_geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
"dst_geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Logstash configuration - Breaking apart src and dst geoip targets.

geoip {
source => "srcip"
database => "/etc/logstash/geo/custom_geoip.dat"
target => "src_geoip"
}
geoip {
source => "dstip"
database => "/etc/logstash/geo/custom_geoip.dat"
target => "dst_geoip"
}

Thanks again!
Cody

13

I can suggest couple of things

  1. Not sure if ES 2.3 support "path": "full" for geoip anymore((remove "path": "full" )), so you just need to do something like following in your template

"dst_geoip" : {"type" : "object","dynamic": true,"properties" : {"location" : { "type" : "geo_point" }}},
"src_geoip" : {"type" : "object","dynamic": true,"properties" : {"location" : { "type" : "geo_point" }}}

  1. When I was using database in logstash 1.5.4 I used following, you need to check if it works for latest LS

geoip {
source => "src"
target => "geoip"
database => "E:\Geo_database\11-02-2016\GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}

  1. Make sure you are using template_overwrite => "true" in LS

Let me know if it works also check this post Problems with geoip configuration

14

geoip {
source => "srcip"
database => "/etc/logstash/geo/custom_geoip.dat"
target => "src_geoip"
add_field => [ "[src_geoip][coordinates]", "%{[src_geoip][longitude]}" ]
add_field => [ "[src_geoip][coordinates]", "%{[src_geoip][latitude]}" ]
}
mutate {
convert => [ "[src_geoip][coordinates]", "float"]
}
geoip {
source => "dstip"
database => "/etc/logstash/geo/custom_geoip.dat"
target => "dst_geoip"
add_field => [ "[dst_geoip][coordinates]", "%{[dst_geoip][longitude]}" ]
add_field => [ "[dst_geoip][coordinates]", "%{[dst_geoip][latitude]}" ]
}
mutate {
convert => [ "[dst_geoip][coordinates]", "float"]
}

Tried to split them this way again. I had to do this also with previous versions of elasticsearch. Still receive the same result. Here is the geoip configuration with the changes. src_geoip coordinates and dst_geoip coordinates still show up as a number are not defined as geopoint on the mapping.

output {
elasticsearch {
hosts => "10.x.x.x"
index => "logstash-fortinet-%{+YYYY.MM.dd}"
template_name => "fortinet*"
template => "/etc/logstash/templates/elasticsearch-fortinet.json"
manage_template => "true"
template_overwrite => "true"
}
}

Template mapping

{
"template" : "fortinet*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date" },
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
},
"dst_geoip" : {"type" : "object","dynamic": true,"properties" : {"location" : { "type" : "geo_point" }}},
"src_geoip" : {"type" : "object","dynamic": true,"properties" : {"location" : { "type" : "geo_point" }}}
}
}
}
}

15

I can get 1 IP geoip filtering to work quite easily but something about overriding this template just has me stumped. Nothing seems to change it in the mapping even with overrides and creating the manual mapping in the json file. Like you I wanted to be able to geohash Tile Map both dst and src addresses.

16

You are creating new indexes after each template update, right? Not expecting existing indexes to change their mappings?

17

Correct. I am clearing the index and just starting fresh each time I change the template. This is a fresh install and I don't plan to start retaining data until I can resolve this issue.

curl -XDELETE http://10.x.x.x:9200/*

18

I got it. Thank you both so much for the help. It was a syntax error on my end. The template name in the logstash configuration did not match the template name identified in the mapping.