# Filters in logstash for ELK

**URL:** <https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995>\
**Category:** Logstash\
**Created:** [August 31, 2017, 11:55am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995 "2017-08-31T11:55:32Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 6, 2017, 3:30pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/21 "2017-09-06T15:30:27Z")

</div>

@ashesh-singh8 yeah, I did check out the mutate filter to do the conversions. My question was why is there a need to use a mutate filter after I've mentioned my data types in the grok pattern. Also, can you share an example of the geo\_ip filter, that's something I haven't looked at. Thanks for the reply, really appreciate it.

---

<div class="post-metadata">

**Author:** ![ashesh-singh8](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ashesh-singh8](https://discuss.elastic.co/u/ashesh-singh8)\
**Post date:** [September 9, 2017, 2:17am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/22 "2017-09-09T02:17:14Z")

</div>

Ok I think I understand you now, I suppose that you are wondering that if you mentioned your data type in `{PATTERN: field}` why field is getting mapped as text always?  
Well you should understand that `PATTERN` is telling you the set of characters which are to be matched which similar to the pattern in a regex expression it does not tell you the data type in which these characters are to be stored.  
For example when you mention the response code as `{NUMBER: response}` it just tells the grok that it has to match characters between 0-9 and save them as text in response.  
That's why you need to convert the data type , now the `{PATTERN: field:type}` is not working for you somehow about which I can't tell you further without having the exact actual case.  
But I think it should be clear to you now that why you need to mutate.

Further you may also wonder that if you mention the data type in index template then why it is not getting parsed correctly so for that you should know that elasticsearch throws an exception for data type mismatch.  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html)

Now about the geo\_ip filter example  
geoip {  
source =\> "ClientIP"  
target =\> "geoip"  
#database =\> "/etc/logstash/GeoLiteCity.dat"  
add\_field =\> ["[geoip][coordinates]", "%{[geoip][longitude]}" ]  
add\_field =\> ["[geoip][coordinates]", "%{[geoip][latitude]}" ]  
}

---

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 18, 2017, 8:42am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/23 "2017-09-18T08:42:31Z")

</div>

@ashesh-singh8 Thanks for the reply and sorry for getting back to you so late. I see your sample, looks promising as well. I just have one issue with that implementation. I have location as a string eg. `location=22.99,75.43` now i don't have separate longitude or latitudes, I can use a kv filter to split the location using a `value_split` where the value is `,` but then how do I assign latitude and longitudes their values?

---

<div class="post-metadata">

**Author:** ![ashesh-singh8](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ashesh-singh8](https://discuss.elastic.co/u/ashesh-singh8)\
**Post date:** [September 18, 2017, 5:52pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/24 "2017-09-18T17:52:52Z")

</div>

Ok. You have the location as string so to implement it you have to do following:-

1. In your elasticsearch index template create a field for storing location , let's say it is `my_loc` with following configuration-

2. In your log stash filter you have the location as a string eg. location="22.99,75.43" . You have to simply add a field `my_loc` and assign it the string-  
add\_field=\>["my\_loc","%{location}"]

You will get your field mapped as geopoint. For more details you can read  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html)

---

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 19, 2017, 6:13am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/25 "2017-09-19T06:13:49Z")

</div>

@ashesh-singh8 that'll fail it I someone simply sends in a `location=22.8774, 77.34353`. Notice the `space` between the `,` and the `77.34353` this will lead to a string with URL encoded log as `location=22.8774,%2077.34353`. So do you have any idea how can i do a URL decoding before I start to even use a kv filter to split my parameters? So that the parameters or rather the raw request doesn't have any unwanted URL encoded fields.

PS: I really appreciate your help on this.

---

<div class="post-metadata">

**Author:** ![ashesh-singh8](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ashesh-singh8](https://discuss.elastic.co/u/ashesh-singh8)\
**Post date:** [September 19, 2017, 12:59pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/26 "2017-09-19T12:59:31Z")

</div>

You can use `gsub` of `mutate` filter to remove the **space** or `%`

```
mutate{ 
     gsub=>["location","[%]",""]
}

```

Notice the space between `[` and `%`.  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub)

---

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 19, 2017, 1:18pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/27 "2017-09-19T13:18:44Z")

</div>

@ashesh-singh8 hi, thanks for the link. Helps out a lot. I just have 1 question, so how do I use the gsub filter to send in the entire request (with url encoding) and get in result the non url encoded request. For eg. All `%20` are replaced with a `space` and all `_` are replaced by a `-` and so on

---

<div class="post-metadata">

**Author:** ![ashesh-singh8](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ashesh-singh8](https://discuss.elastic.co/u/ashesh-singh8)\
**Post date:** [September 19, 2017, 2:47pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/28 "2017-09-19T14:47:43Z")

</div>

To get the whole request urldecoded use `urldecode` filter plugin

> **[Urldecode filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-urldecode.html)**

You can also see this thread for a ruby implementation

> [@How to urldecode %uXXXX type of strings?](https://discuss.elastic.co/t/how-to-urldecode-uxxxx-type-of-strings/27718):
>
> I needed a javascript equivalent of unescape in logstash. I know %uXXXX is not standard but this is still quite common. A lot of our users' browsers send urls in this format. I am reserving the use of ruby filter in logstash to decode this type of url as the last resort. Before that, I am looking for a less heavy solution. Thanks for any suggestions.

After this if any specific character you need to replace then apply separate gsub for each.

---

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 19, 2017, 3:19pm UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/29 "2017-09-19T15:19:12Z")

</div>

@ashesh-singh8 Great sounds great. I have 1 last question, Can I nest multiple statements in a mutate filter?

```
mutate {
    add_field => { "search_query" => "%{keyword}" }
    add_field => { "search_query" => "%{keyword}" }
}

```

When I put the above code filter code, logstash passed the config test but the data didn't get pushed and when I modified it to the below code, it started working.

```
mutate {
    add_field => { "search_query" => "%{keyword}" }
}
mutate {
       add_field => { "search_query" => "%{keyword}" }
}

```

Is there anything specific about this?

---

<div class="post-metadata">

**Author:** ![ashesh-singh8](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ashesh-singh8](https://discuss.elastic.co/u/ashesh-singh8)\
**Post date:** [September 20, 2017, 4:40am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/30 "2017-09-20T04:40:03Z")

</div>

Look

> <https://github.com/logstash-plugins/logstash-filter-mutate/issues/27>

> <https://github.com/logstash-plugins/logstash-filter-mutate/issues/28>

---

<div class="post-metadata">

**Author:** ![iam.Carrot](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\
**Post date:** [September 20, 2017, 4:56am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/31 "2017-09-20T04:56:00Z")

</div>

Oh sweet. Thanks a lot. you've been of great help:)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 18, 2017, 4:56am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995/32 "2017-10-18T04:56:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995.md?page=1)