How to write custom gork for logstash


(Rocky Rk) #1

I'm trying to test the some custom log filter for logstash but somehow i'm not able to get it, I googled and looked over many examples but I am not able to create a one I want.

Below is my log patterns:

testhost-in2,19/01/11,06:34,04-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /test/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-
testhost-in2,19/01/11,06:40,09-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\
testhost-in2,19/01/11,06:45,14-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:01 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\

below is what I trying to create but does not works.

HOSTNAME:hostname  DATE:date HOURS_Time:hour_min  USERNAME: username USERNAME:username  NUMBER:pid   COMMAND

any help Will be much appreciated.


(Christian Dahlqvist) #2

The grok pattern must exactly match the data and your fields seem to be separated by comma, and not spaces as used in the grok pattern. I would recommend reading this introductory blog post, which describes how to build out grok patterns in a systematic way.


(Rocky Rk) #3

@Christian_Dahlqvist , thanks for your revert, I tried the below and it works but there seems more tweek.

%{HOSTNAME:hostname},%{DATE:date},%{HOUR:hour1}:%{MINUTE:minute1},%{NUMBER}-%{WORD},%{USER:user},%{USER:user2} %{NUMBER:pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:number1} %{NUMBER:number2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}


#4

You say it works but do not say what you do not like about the results.


(Rocky Rk) #5

@Badger, thanks for your response,

I have below log :

aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

So, When i use the provided grok it turns the below output on the groker, While using it on the logstash and plotting on the Kibana the PATH & UNIXPATH prefixes the location of the as such/home/ELK/scratch/ptr/aruba.20974rather it should be just/scratch/ptr/aruba.20974` .

{
  "hostname": [
    [
      "aruba"
    ]
  ],
  "date": [
    [
      "19/01/02"
    ]
  ],
  "DATE_US": [
    [
      null
    ]
  ],
  "MONTHNUM": [
    [
      null,
      "01"
    ]
  ],
  "MONTHDAY": [
    [
      null,
      "19"
    ]
  ],
  "YEAR": [
    [
      null,
      "02"
    ]
  ],
  "DATE_EU": [
    [
      "19/01/02"
    ]
  ],
  "hour1": [
    [
      "05"
    ]
  ],
  "minute1": [
    [
      "14"
    ]
  ],
  "NUMBER": [
    [
      "00"
    ]
  ],
  "BASE10NUM": [
    [
      "00",
      "120219",
      "0.0",
      "0.0",
      "9252",
      "1164"
    ]
  ],
  "WORD": [
    [
      "mins"
    ]
  ],
  "user": [
    [
      "ntester"
    ]
  ],
  "USERNAME": [
    [
      "ntester",
      "ntester"
    ]
  ],
  "user2": [
    [
      "ntester"
    ]
  ],
  "pid": [
    [
      "120219"
    ]
  ],
  "float": [
    [
      "0.0",
      "0.0"
    ]
  ],
  "number1": [
    [
      "9252"
    ]
  ],
  "number2": [
    [
      "1164"
    ]
  ],
  "DATA": [
    [
      "pts/1 S+"
    ]
  ],
  "hour2": [
    [
      "05"
    ]
  ],
  "minute2": [
    [
      "14"
    ]
  ],
  "hour3": [
    [
      "0"
    ]
  ],
  "minute3": [
    [
      "00"
    ]
  ],
  "command": [
    [
      "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1"
    ]
  ],
  "path": [
    [
      "/scratch/ptr/aruba.20974"
    ]
  ],
  "UNIXPATH": [
    [
      "/scratch/ptr/aruba.20974"
    ]
  ],
  "WINPATH": [
    [
      null
    ]
  ]
}

#6

The pattern you showed does not use UNIXPATH (and not using UNIXPATH is a good thing). So I don't see how it could produce that data. And I do not believe anything in the Elastic stack is spontaneously prefixing a string with /home/ELK.

Can you show a working configuration, the input and output?


(Rocky Rk) #7

My Working configuration:

$ cat /etc/logstash/conf.d/logstash-log.conf

input {
  file {
    path => [ "/home/elk/*.txt" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "rmlog"
  }
}

filter {
  if [type] == "rmlog" {
    grok {
      match => { "message" => "%{HOSTNAME:Hostname},%{DATE:Date},%{HOUR:Hour_since}:%{MINUTE:Mins_since},%{NUMBER}-%{WORD},%{USER:User_1},%{USER:User_2} %{NUMBER:Pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:Num_1} %{NUMBER:Num_2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      
   }
 }
}

output {
        if [type] == "rmlog" {
        elasticsearch {
                hosts => ["my-elk:9200"]
                manage_template => false
                index => "rmlog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Main Problem here as it takes only the First part of the log Just before comma(,) and thereafter it does not takes anything..

`

aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

`

In the above line it takes only the /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1 but does not take /scratch/ptr/aruba.20974 .

So, my concern is to capture anything starting ^rm or ^/bin/rm, os say immediately after seeing rm i need to have everything including it.


#8

With that configuration and log line I get

       "path" => "/scratch/ptr/aruba.20974",
    "command" => "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1",

I can't think why you wouldn't get the same.


(Rocky Rk) #9

That's true it works but on the Kibana portal it splits into two parts as you see, while i need it into one line itself like:

/bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

I got it into one line while removing the PATH from grok after GREEDYDATA