# Can someone please help me with Geo filter on said logstash conf

**URL:** <https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366>\
**Category:** Logstash\
**Created:** [August 25, 2017, 12:42pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366 "2017-08-25T12:42:51Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 12:42pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/1 "2017-08-25T12:42:51Z")

</div>

Hi Guys,

I am not expert in logstash and I need help regarding indexing my opencanary data with geo tags. I already have below conf but I am sure it does not have geo tag.

Can someone please help me with appropriate config.

input {  
file {  
path =\> "/var/log/opencanary.log"  
start\_position =\> "beginning"  
}  
}

filter {  
mutate {  
add\_tag =\> ["opencanary"]  
}  
json {  
source =\> "message"  
target =\> "parsedJson"  
}  
mutate {  
add\_field =\> {  
"dst\_host" =\> "%{[parsedJson][dst\_host]}"  
"dst\_port" =\> "%{[parsedJson][dst\_port]}"  
"local\_time" =\> "%{[parsedJson][local\_time]}"  
"logdata" =\> "%{[parsedJson][logdata]}"  
"logtype" =\> "%{[parsedJson][logtype]}"  
"node\_id" =\> "%{[parsedJson][node\_id]}"  
"src\_host" =\> "%{[parsedJson][src\_host]}"  
"src\_port" =\> "%{[parsedJson][src\_port]}"  
}  
remove\_field =\> ["parsedJson","message"]  
}  
json {  
source =\> "logdata"  
target =\> "parsedlogdata"  
}  
mutate {  
add\_field =\> {  
"SESSION" =\> "%{[parsedlogdata][SESSION]}"  
}  
remove\_field =\> ["path","@version","parsedlogdata.SESSION"]  
}

# grok {

# match =\> ["local\_time", "%{DATESTAMP:timestamp}"]

# }

```
    date {
            match => ["local_time", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
            timezone => "Asia/Kolkata"
            remove_field => "timestamp"
    }

```

}

output {  
elasticsearch {  
manage\_template =\> false  
hosts =\> "192.168.5.15:9200"  
index =\> "logstash-opencanary-%{+YYYY.MM.dd}"  
}  
stdout {  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2017, 1:27pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/2 "2017-08-25T13:27:20Z")

</div>

And what does the log you're attempting to parse look like?

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 1:39pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/3 "2017-08-25T13:39:33Z")

</div>

Geo Map from source addresses and heat map according to source IP addresses. And here are the logs examples

{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 10:59:15.939362", "logdata": {"PASSWORD": "xmhdipc", "USERNAME": "root"}, "lo  
gtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 54656}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 10:59:46.990790", "logdata": {"PASSWORD": "1234", "USERNAME": "admin"}, "logt  
ype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 55507}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:00:17.880832", "logdata": {"PASSWORD": "support", "USERNAME": "support"},  
"logtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 56282}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:00:48.976065", "logdata": {"PASSWORD": "12345", "USERNAME": "root"}, "logt  
ype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 57009}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:01:19.916128", "logdata": {"PASSWORD": "1001chin", "USERNAME": "root"}, "l  
ogtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 39636}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:01:50.909242", "logdata": {"PASSWORD": "password", "USERNAME": "root"}, "l  
ogtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 40404}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": true, "local\_time": "2017-07-24 11:02:21.933574", "logdata": {"PASSWORD": "password", "USERNAME": "admin"}, "l  
ogtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 41249}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:02:52.982781", "logdata": {"PASSWORD": "54321", "USERNAME": "root"}, "logt  
ype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 42182}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:03:23.914468", "logdata": {"PASSWORD": "juantech", "USERNAME": "root"}, "l  
ogtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 42988}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:03:54.957853", "logdata": {"PASSWORD": "admin", "USERNAME": "root"}, "logt  
ype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 43776}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:04:25.957863", "logdata": {"PASSWORD": "12345", "USERNAME": "admin"}, "log  
type": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 44499}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:04:56.920228", "logdata": {"PASSWORD": "123456", "USERNAME": "admin"}, "lo  
gtype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 45144}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:05:27.886476", "logdata": {"PASSWORD": "1111", "USERNAME": "admin"}, "logt  
ype": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 46026}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:05:58.885839", "logdata": {"PASSWORD": "Zte521", "USERNAME": "root"}, "log  
type": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 46970}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:06:29.984932", "logdata": {"PASSWORD": "user", "USERNAME": "user"}, "logty  
pe": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 59809}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:07:00.923142", "logdata": {"PASSWORD": "guest", "USERNAME": "guest"}, "log  
type": 6001, "node\_id": "opencanary-1", "src\_host": "91.193.69.190", "src\_port": 60809}  
{"dst\_host": "172.31.43.40", "dst\_port": 23, "honeycred": false, "local\_time": "2017-07-24 11:07:31.926856", "logdata": {"PASSWORD": "zlxx.", "USERNAME": "root"}, "logt

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2017, 2:04pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/4 "2017-08-25T14:04:40Z")

</div>

Have you tried adding a geoip filter to look up one of the IP address fields?

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 2:06pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/5 "2017-08-25T14:06:23Z")

</div>

Would you please give me example? Though I am going though the logs

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 2:36pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/6 "2017-08-25T14:36:58Z")

</div>

Ok - I tried with below setup is that OK

```
    geoip {
            source => "src_host"
            add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ]

            }

```

}

filter {  
mutate {  
add\_tag =\> ["opencanary"]  
}  
json {  
source =\> "message"  
target =\> "parsedJson"  
}  
mutate {  
add\_field =\> {  
"dst\_host" =\> "%{[parsedJson][dst\_host]}"  
"dst\_port" =\> "%{[parsedJson][dst\_port]}"  
"local\_time" =\> "%{[parsedJson][local\_time]}"  
"logdata" =\> "%{[parsedJson][logdata]}"  
"logtype" =\> "%{[parsedJson][logtype]}"  
"node\_id" =\> "%{[parsedJson][node\_id]}"  
"src\_host" =\> "%{[parsedJson][src\_host]}"  
"src\_port" =\> "%{[parsedJson][src\_port]}"  
}  
remove\_field =\> ["parsedJson","message"]

But my query is how do I pass index which is already done? DO I need to delete that from kibana and index it again?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2017, 8:34pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/7 "2017-08-25T20:34:54Z")

</div>

Why are you storing the lon/lat into `[geoip][coordinates]`? Logstash's default index template for ES maps the `[geoip][location]` field as geo\_point.

It's not clear where you've placed your geoip filter. Presumably after the mutate filter as the `src_host` field won't exist otherwise.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 26, 2017, 2:32am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/8 "2017-08-26T02:32:17Z")

</div>

OK I changed my filter but still Geoip is not getting indexed..here is complete config file

input {  
file {  
path =\> "/var/log/opencanary.log"  
start\_position =\> "beginning"  
}  
}

filter {  
mutate {  
add\_tag =\> ["opencanary"]  
}  
json {  
source =\> "message"  
target =\> "parsedJson"  
}  
mutate {  
add\_field =\> {  
"dst\_host" =\> "%{[parsedJson][dst\_host]}"  
"dst\_port" =\> "%{[parsedJson][dst\_port]}"  
"local\_time" =\> "%{[parsedJson][local\_time]}"  
"logdata" =\> "%{[parsedJson][logdata]}"  
"logtype" =\> "%{[parsedJson][logtype]}"  
"node\_id" =\> "%{[parsedJson][node\_id]}"  
"src\_host" =\> "%{[parsedJson][src\_host]}"  
"src\_port" =\> "%{[parsedJson][src\_port]}"  
}  
remove\_field =\> ["parsedJson","message"]  
}

```
    geoip {
            source => "src_host"

            }

    json {
            source => "logdata"
            target => "parsedlogdata"
    }
    mutate {
            add_field => {
                    "SESSION" => "%{[parsedlogdata][SESSION]}"
            }
            remove_field => ["path","@version","parsedlogdata.SESSION"]
    }

```

# grok {

# match =\> ["local\_time", "%{DATESTAMP:timestamp}"]

# }

```
    date {
            match => ["local_time", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
            timezone => "Asia/Kolkata"
            remove_field => "timestamp"
    }

```

}

output {  
elasticsearch {  
manage\_template =\> false  
hosts =\> "192.168.5.15:9200"  
index =\> "logstash-opencanary-%{+YYYY.MM.dd}"  
}  
stdout {  
codec =\> rubydebug  
}  
}

Not sure if the index already exists is that the reason geoip is not being reflected? In that case do I need delete the index and recreate one or refresh it?

I tried refreshing couple of time but no luck.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 28, 2017, 7:12am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/9 "2017-08-28T07:12:24Z")

</div>

- What does an example event look like? Show the output from your stdout output or copy/paste from Kibana's JSON tab.
- What do the mappings of your index look like? Use ES's get mapping API.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2017, 7:12am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-geo-filter-on-said-logstash-conf/98366/10 "2017-09-25T07:12:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.