How to write custom gork for logstash

Elastic Stack Logstash
1

I'm trying to test the some custom log filter for logstash but somehow i'm not able to get it, I googled and looked over many examples but I am not able to create a one I want.

Below is my log patterns:

testhost-in2,19/01/11,06:34,04-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /test/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-
testhost-in2,19/01/11,06:40,09-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\
testhost-in2,19/01/11,06:45,14-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:01 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119-\

below is what I trying to create but does not works.

HOSTNAME:hostname  DATE:date HOURS_Time:hour_min  USERNAME: username USERNAME:username  NUMBER:pid   COMMAND

any help Will be much appreciated.

2

The grok pattern must exactly match the data and your fields seem to be separated by comma, and not spaces as used in the grok pattern. I would recommend reading this introductory blog post, which describes how to build out grok patterns in a systematic way.

3

@Christian_Dahlqvist , thanks for your revert, I tried the below and it works but there seems more tweek.

%{HOSTNAME:hostname},%{DATE:date},%{HOUR:hour1}:%{MINUTE:minute1},%{NUMBER}-%{WORD},%{USER:user},%{USER:user2} %{NUMBER:pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:number1} %{NUMBER:number2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}

4

You say it works but do not say what you do not like about the results.

1 Like
5

@Badger, thanks for your response,

I have below log :

aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

So, When i use the provided grok it turns the below output on the groker, While using it on the logstash and plotting on the Kibana the PATH & UNIXPATH prefixes the location of the as such/home/ELK/scratch/ptr/aruba.20974 rather it should be just /scratch/ptr/aruba.20974` .

{
  "hostname": [
    [
      "aruba"
    ]
  ],
  "date": [
    [
      "19/01/02"
    ]
  ],
  "DATE_US": [
    [
      null
    ]
  ],
  "MONTHNUM": [
    [
      null,
      "01"
    ]
  ],
  "MONTHDAY": [
    [
      null,
      "19"
    ]
  ],
  "YEAR": [
    [
      null,
      "02"
    ]
  ],
  "DATE_EU": [
    [
      "19/01/02"
    ]
  ],
  "hour1": [
    [
      "05"
    ]
  ],
  "minute1": [
    [
      "14"
    ]
  ],
  "NUMBER": [
    [
      "00"
    ]
  ],
  "BASE10NUM": [
    [
      "00",
      "120219",
      "0.0",
      "0.0",
      "9252",
      "1164"
    ]
  ],
  "WORD": [
    [
      "mins"
    ]
  ],
  "user": [
    [
      "ntester"
    ]
  ],
  "USERNAME": [
    [
      "ntester",
      "ntester"
    ]
  ],
  "user2": [
    [
      "ntester"
    ]
  ],
  "pid": [
    [
      "120219"
    ]
  ],
  "float": [
    [
      "0.0",
      "0.0"
    ]
  ],
  "number1": [
    [
      "9252"
    ]
  ],
  "number2": [
    [
      "1164"
    ]
  ],
  "DATA": [
    [
      "pts/1 S+"
    ]
  ],
  "hour2": [
    [
      "05"
    ]
  ],
  "minute2": [
    [
      "14"
    ]
  ],
  "hour3": [
    [
      "0"
    ]
  ],
  "minute3": [
    [
      "00"
    ]
  ],
  "command": [
    [
      "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1"
    ]
  ],
  "path": [
    [
      "/scratch/ptr/aruba.20974"
    ]
  ],
  "UNIXPATH": [
    [
      "/scratch/ptr/aruba.20974"
    ]
  ],
  "WINPATH": [
    [
      null
    ]
  ]
}
6

The pattern you showed does not use UNIXPATH (and not using UNIXPATH is a good thing). So I don't see how it could produce that data. And I do not believe anything in the Elastic stack is spontaneously prefixing a string with /home/ELK.

Can you show a working configuration, the input and output?

7

My Working configuration:

$ cat /etc/logstash/conf.d/logstash-log.conf

input {
  file {
    path => [ "/home/elk/*.txt" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "rmlog"
  }
}

filter {
  if [type] == "rmlog" {
    grok {
      match => { "message" => "%{HOSTNAME:Hostname},%{DATE:Date},%{HOUR:Hour_since}:%{MINUTE:Mins_since},%{NUMBER}-%{WORD},%{USER:User_1},%{USER:User_2} %{NUMBER:Pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:Num_1} %{NUMBER:Num_2} %{DATA} %{HOUR:hour2}:%{MINUTE:minute2} %{HOUR:hour3}:%{MINUTE:minute3} %{GREEDYDATA:command},%{PATH:path}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      
   }
 }
}

output {
        if [type] == "rmlog" {
        elasticsearch {
                hosts => ["my-elk:9200"]
                manage_template => false
                index => "rmlog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Main Problem here as it takes only the First part of the log Just before comma(,) and thereafter it does not takes anything..

`

aruba,19/01/02,05:14,00-mins,ntester,ntester 120219 0.0 0.0 9252 1164 pts/1 S+ 05:14 0:00 sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

`

In the above line it takes only the /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1 but does not take /scratch/ptr/aruba.20974 .

So, my concern is to capture anything starting ^rm or ^/bin/rm, os say immediately after seeing rm i need to have everything including it.

8

With that configuration and log line I get

       "path" => "/scratch/ptr/aruba.20974",
    "command" => "sh -c /bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1",

I can't think why you wouldn't get the same.

9

That's true it works but on the Kibana portal it splits into two parts as you see, while i need it into one line itself like:

/bin/rm -rf /scratch/ptr/aruba.20974/ptr_tmp/sn.aruba.25179.1546385887 > /dev/null 2>&1,/scratch/ptr/aruba.20974

I got it into one line while removing the PATH from grok after GREEDYDATA

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.