Generic advice to a newb. Best practises

Elastic Stack Logstash
1

I have just installed ELK, got my first data into to it, done a bit of grokking and "kv"..ing. I am however an absolute noob and the problem is more complex than can be solved with this. So what I am looking for is generic advice on the best way to proceed with complex logs. you can take a look at sample logs if you so wish. key values don't really work in all cases. I googled around and found this construct

grok {break_on_match => true
match => ["message", "pattern1",
"message", "pattern2",
"message", "pattern3"
] }
followed by if [message] =~ sequences to further parse the nonsense from sophos :wink:

Is this a sensible approach or are there better ways?

2

Why doesn't grok + kv work in this case?

3

for example, on some of the WAF type entries, urls containing "=" characters get chopped up into random key / values. ( that was my first attempt!) so I added an include_keys which improved that handling but I still ended up with situations where if a value for a field was missing, so i might have ' dst_country_code= dst_ip="8.8.8.8" ' and the dst_ip would be dropped. But I am not asking for specific solutions, just a pointer as to what the recommended way of dealing with messy log formats. ie is using some kind of if / else construct the right way to do this . I am going to persuade the boss to spring for some of the online courses which should reduce the number of dumb questions from me :slight_smile:

4

It looks like there are issues related to some of the problems you are facing around parsing data with empty values and handling values with separator character in the value.