# Generic advice to a newb. Best practises

**URL:** <https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469>\
**Category:** Logstash\
**Created:** [April 6, 2017, 12:44pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469 "2017-04-06T12:44:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bhu73](https://avatars.discourse-cdn.com/v4/letter/b/7ea924/32.png) [@bhu73](https://discuss.elastic.co/u/bhu73)\
**Post date:** [April 6, 2017, 12:44pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469/1 "2017-04-06T12:44:48Z")

</div>

I have just installed ELK, got my first data into to it, done a bit of grokking and "kv"..ing. I am however an absolute noob and the problem is more complex than can be solved with this. So what I am looking for is generic advice on the best way to proceed with complex logs. you can take a look at [sample logs](http://sp.mine.nu/logs.html) if you so wish. key values don't really work in all cases. I googled around and found this construct

grok {break\_on\_match =\> true  
match =\> ["message", "pattern1",  
"message", "pattern2",  
"message", "pattern3"  
] }  
followed by if [message] =~ sequences to further parse the nonsense from sophos 😉

Is this a sensible approach or are there better ways?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 6, 2017, 12:47pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469/2 "2017-04-06T12:47:27Z")

</div>

Why doesn't grok + kv work in this case?

---

<div class="post-metadata">

**Author:** ![bhu73](https://avatars.discourse-cdn.com/v4/letter/b/7ea924/32.png) [@bhu73](https://discuss.elastic.co/u/bhu73)\
**Post date:** [April 6, 2017, 1:10pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469/3 "2017-04-06T13:10:09Z")

</div>

for example, on some of the WAF type entries, urls containing "=" characters get chopped up into random key / values. ( that was my first attempt!) so I added an include\_keys which improved that handling but I still ended up with situations where if a value for a field was missing, so i might have ' dst\_country\_code= dst\_ip="8.8.8.8" ' and the dst\_ip would be dropped. But I am not asking for specific solutions, just a pointer as to what the recommended way of dealing with messy log formats. ie is using some kind of if / else construct the right way to do this . I am going to persuade the boss to spring for some of the online courses which should reduce the number of dumb questions from me 🙂

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469/4 "2017-04-06T13:26:39Z")

</div>

It looks like there are issues related to some of the problems you are facing around [parsing data with empty values](https://github.com/logstash-plugins/logstash-filter-kv/issues/37) and [handling values with separator character in the value](https://github.com/logstash-plugins/logstash-filter-kv/issues/30).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2017, 1:26pm UTC](https://discuss.elastic.co/t/generic-advice-to-a-newb-best-practises/81469/5 "2017-05-04T13:26:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.