Generic advice to a newb. Best practises

I have just installed ELK, got my first data into to it, done a bit of grokking and "kv"..ing. I am however an absolute noob and the problem is more complex than can be solved with this. So what I am looking for is generic advice on the best way to proceed with complex logs. you can take a look at sample logs if you so wish. key values don't really work in all cases. I googled around and found this construct

grok {break_on_match => true
match => ["message", "pattern1",
"message", "pattern2",
"message", "pattern3"
] }
followed by if [message] =~ sequences to further parse the nonsense from sophos :wink:

Is this a sensible approach or are there better ways?

Why doesn't grok + kv work in this case?

for example, on some of the WAF type entries, urls containing "=" characters get chopped up into random key / values. ( that was my first attempt!) so I added an include_keys which improved that handling but I still ended up with situations where if a value for a field was missing, so i might have ' dst_country_code= dst_ip="8.8.8.8" ' and the dst_ip would be dropped. But I am not asking for specific solutions, just a pointer as to what the recommended way of dealing with messy log formats. ie is using some kind of if / else construct the right way to do this . I am going to persuade the boss to spring for some of the online courses which should reduce the number of dumb questions from me :slight_smile:

It looks like there are issues related to some of the problems you are facing around parsing data with empty values and handling values with separator character in the value.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.