Filters in logstash for ELK

Elastic Stack Logstash
21

@ashesh-singh8 yeah, I did check out the mutate filter to do the conversions. My question was why is there a need to use a mutate filter after I've mentioned my data types in the grok pattern. Also, can you share an example of the geo_ip filter, that's something I haven't looked at. Thanks for the reply, really appreciate it.

22

Ok I think I understand you now, I suppose that you are wondering that if you mentioned your data type in {PATTERN: field} why field is getting mapped as text always?
Well you should understand that PATTERN is telling you the set of characters which are to be matched which similar to the pattern in a regex expression it does not tell you the data type in which these characters are to be stored.
For example when you mention the response code as {NUMBER: response} it just tells the grok that it has to match characters between 0-9 and save them as text in response.
That's why you need to convert the data type , now the {PATTERN: field:type} is not working for you somehow about which I can't tell you further without having the exact actual case.
But I think it should be clear to you now that why you need to mutate.

Further you may also wonder that if you mention the data type in index template then why it is not getting parsed correctly so for that you should know that elasticsearch throws an exception for data type mismatch.
https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html

Now about the geo_ip filter example
geoip {
source => "ClientIP"
target => "geoip"
#database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

23

@ashesh-singh8 Thanks for the reply and sorry for getting back to you so late. I see your sample, looks promising as well. I just have one issue with that implementation. I have location as a string eg. location=22.99,75.43 now i don't have separate longitude or latitudes, I can use a kv filter to split the location using a value_split where the value is , but then how do I assign latitude and longitudes their values?

24

Ok. You have the location as string so to implement it you have to do following:-

  1. In your elasticsearch index template create a field for storing location , let's say it is my_loc with following configuration-

    "mappings": {
    "my_type": {
    "properties": {
    "my_loc": {
    "type": "geo_point"
    }
    }
    }
    }

  2. In your log stash filter you have the location as a string eg. location="22.99,75.43" . You have to simply add a field my_loc and assign it the string-
    add_field=>["my_loc","%{location}"]

You will get your field mapped as geopoint. For more details you can read
https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

25

@ashesh-singh8 that'll fail it I someone simply sends in a location=22.8774, 77.34353. Notice the space between the , and the 77.34353 this will lead to a string with URL encoded log as location=22.8774,%2077.34353. So do you have any idea how can i do a URL decoding before I start to even use a kv filter to split my parameters? So that the parameters or rather the raw request doesn't have any unwanted URL encoded fields.

PS: I really appreciate your help on this.

26

You can use gsub of mutate filter to remove the space or %

mutate{ 
     gsub=>["location","[ %]",""]
}

Notice the space between [ and %.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub

27

@ashesh-singh8 hi, thanks for the link. Helps out a lot. I just have 1 question, so how do I use the gsub filter to send in the entire request (with url encoding) and get in result the non url encoded request. For eg. All %20 are replaced with a space and all _ are replaced by a - and so on

28

To get the whole request urldecoded use urldecode filter plugin

You can also see this thread for a ruby implementation

After this if any specific character you need to replace then apply separate gsub for each.

29

@ashesh-singh8 Great sounds great. I have 1 last question, Can I nest multiple statements in a mutate filter?

mutate {
    add_field => { "search_query" => "%{keyword}" }
    add_field => { "search_query" => "%{keyword}" }
}

When I put the above code filter code, logstash passed the config test but the data didn't get pushed and when I modified it to the below code, it started working.

mutate {
    add_field => { "search_query" => "%{keyword}" }
}
mutate {
       add_field => { "search_query" => "%{keyword}" }
}

Is there anything specific about this?

30

Look

31

Oh sweet. Thanks a lot. you've been of great help:)

32

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.