# How to write custom gork for logstash **URL:** \ **Category:** Logstash\ **Created:** [January 11, 2019, 8:04am UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858 "2019-01-11T08:04:00Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Rocky\_RK](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Post date:** [January 11, 2019, 8:04am UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/1 "2019-01-11T08:04:00Z")
I'm trying to test the some custom log filter for logstash but somehow i'm not able to get it, I googled and looked over many examples but I am not able to create a one I want. **Below is my log patterns:** ``` testhost-in2,19/01/11,06:34,04-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /test/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119- testhost-in2,19/01/11,06:40,09-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\ testhost-in2,19/01/11,06:45,14-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:01 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\ ``` below is what I trying to create but does not works. ``` HOSTNAME:hostname DATE:date HOURS_Time:hour_min USERNAME: username USERNAME:username NUMBER:pid COMMAND ``` any help Will be much appreciated. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 13, 2019, 12:01pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/2 "2019-01-13T12:01:48Z")
The grok pattern must exactly match the data and your fields seem to be separated by comma, and not spaces as used in the grok pattern. I would recommend reading [this introductory blog post](https://www.elastic.co/blog/a-practical-introduction-to-logstash), which describes how to build out grok patterns in a systematic way. ---
**Author:** ![Rocky\_RK](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Post date:** [January 13, 2019, 2:51pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/3 "2019-01-13T14:51:24Z")
@Christian_Dahlqvist , thanks for your revert, I tried the below and it works but there seems more tweek. `%{HOSTNAME:hostname},%{DATE:date},%{HOUR:hour1}:%{MINUTE:minute1},%{NUMBER}-%{WORD},%{USER:user},%{USER:user2} %{NUMBER:pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:number1} %{NUMBER:number2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 13, 2019, 3:19pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/4 "2019-01-13T15:19:56Z")
You say it works but do not say what you do not like about the results. ---
**Author:** ![Rocky\_RK](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Post date:** [January 13, 2019, 4:39pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/5 "2019-01-13T16:39:15Z")
@Badger, thanks for your response, I have below log : > aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr\_tmp/sn.aruba.25179.1546385887 \> /dev/null 2\>&1,/scratch/ptr/aruba.20974 So, When i use the provided grok it turns the below output on the groker, While using it on the logstash and plotting on the Kibana the PATH & UNIXPATH prefixes the location of the `as such`/home/ELK/scratch/ptr/aruba.20974` rather it should be just `/scratch/ptr/aruba.20974` . ``` { "hostname": [ [ "aruba" ] ], "date": [ [ "19/01/02" ] ], "DATE_US": [ [ null ] ], "MONTHNUM": [ [ null, "01" ] ], "MONTHDAY": [ [ null, "19" ] ], "YEAR": [ [ null, "02" ] ], "DATE_EU": [ [ "19/01/02" ] ], "hour1": [ [ "05" ] ], "minute1": [ [ "14" ] ], "NUMBER": [ [ "00" ] ], "BASE10NUM": [ [ "00", "120219", "0.0", "0.0", "9252", "1164" ] ], "WORD": [ [ "mins" ] ], "user": [ [ "ntester" ] ], "USERNAME": [ [ "ntester", "ntester" ] ], "user2": [ [ "ntester" ] ], "pid": [ [ "120219" ] ], "float": [ [ "0.0", "0.0" ] ], "number1": [ [ "9252" ] ], "number2": [ [ "1164" ] ], "DATA": [ [ "pts/1 S+" ] ], "hour2": [ [ "05" ] ], "minute2": [ [ "14" ] ], "hour3": [ [ "0" ] ], "minute3": [ [ "00" ] ], "command": [ [ "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1" ] ], "path": [ [ "/scratch/ptr/aruba.20974" ] ], "UNIXPATH": [ [ "/scratch/ptr/aruba.20974" ] ], "WINPATH": [ [ null ] ] } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 13, 2019, 5:23pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/6 "2019-01-13T17:23:28Z")
The pattern you showed does not use UNIXPATH (and not using UNIXPATH is a [good thing](https://github.com/logstash-plugins/logstash-patterns-core/issues/159)). So I don't see how it could produce that data. And I do not believe anything in the Elastic stack is spontaneously prefixing a string with /home/ELK. Can you show a working configuration, the input and output? ---
**Author:** ![Rocky\_RK](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Post date:** [January 15, 2019, 5:59am UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/7 "2019-01-15T05:59:52Z")
My Working configuration: ``` $ cat /etc/logstash/conf.d/logstash-log.conf input { file { path => ["/home/elk/*.txt"] start_position => beginning sincedb_path => "/dev/null" type => "rmlog" } } filter { if [type] == "rmlog" { grok { match => { "message" => "%{HOSTNAME:Hostname},%{DATE:Date},%{HOUR:Hour_since}:%{MINUTE:Mins_since},%{NUMBER}-%{WORD},%{USER:User_1},%{USER:User_2} %{NUMBER:Pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:Num_1} %{NUMBER:Num_2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}" } add_field => ["received_at", "%{@timestamp}"] } } } output { if [type] == "rmlog" { elasticsearch { hosts => ["my-elk:9200"] manage_template => false index => "rmlog-%{+YYYY.MM.dd}" document_type => "messages" } } } ``` Main Problem here as it takes only the First part of the log Just before comma(,) and thereafter it does not takes anything.. ` > aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr\_tmp/sn.aruba.25179.1546385887 \> /dev/null 2\>&1,/scratch/ptr/aruba.20974 ` In the above line it takes only the `/bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1` but does not take `/scratch/ptr/aruba.20974` . So, my concern is to capture anything starting `^rm` or `^/bin/rm`, os say immediately after seeing rm i need to have everything including it. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 15, 2019, 1:31pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/8 "2019-01-15T13:31:59Z")
With that configuration and log line I get ``` "path" => "/scratch/ptr/aruba.20974", "command" => "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1", ``` I can't think why you wouldn't get the same. ---
**Author:** ![Rocky\_RK](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Post date:** [January 15, 2019, 5:37pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/9 "2019-01-15T17:37:28Z")
That's true it works but on the Kibana portal it splits into two parts as you see, while i need it into one line itself like: ``` /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974 ``` I got it into one line while removing the `PATH` from grok after `GREEDYDATA` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2019, 5:37pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858/10 "2019-02-12T17:37:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.