Logstash pipeline order

Hello All,

Is there any order of logstash pipelines config: I've below pipelines:

[root@gzp-p-qv-logst-3 pipeline]# ls -larth
total 60K
drwxr-xr-x. 3 logstash logstash   42 Oct 11  2019 ..
-rw-r--r--. 1 root     root       99 Nov 11  2019 01-input.conf.j2
-rw-r--r--. 1 root     root      784 Nov 11  2019 50-auditbeat.conf.j2
-rw-r--r--. 1 root     root     1019 Nov 11  2019 50-cassandra.conf.j2
-rw-r--r--. 1 root     root      963 Nov 11  2019 50-httperror.conf.j2
-rw-r--r--. 1 root     root      799 Nov 11  2019 50-kerberos.conf.j2
-rw-r--r--. 1 root     root     4.5K Nov 11  2019 50-platform.conf.j2
-rw-r--r--. 1 root     root     3.6K Nov 11  2019 50-realm.conf.j2
-rw-r--r--. 1 root     root      565 Nov 11  2019 50-secure.conf.j2
-rw-r--r--. 1 root     root      372 Nov 11  2019 98-default_indexes.conf.j2
-rw-r--r--. 1 root     root     2.4K Nov 13  2019 97-outputlogstash.conf
-rw-r--r--. 1 root     root      757 Mar  2 01:41 49-nginx.conf.j2
-rw-r--r--. 1 root     root     2.3K Apr  1 18:35 99-output.conf
-rw-r--r--. 1 root     root      566 Apr  1 19:04 51-applog.conf.j2
drwxr-xr-x. 2 root     root     4.0K Apr  1 21:52 .
[root@gzp-p-qv-logst-3 pipeline]# 

I've created 51-applog.conf.j2 like below:

filter {
  if ("qapi" in [tags])
   {
    #mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}
    add_field => [ "tag", "applog" ]
    }

    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }

   mutate { remove_field => [ "host", "message", "@version","timestamp" ] }
  }

}

But seems not working.....
Please help me in understanding the order of pipeline and naming conventions.

Regards.
Banik

First of all, looking to the logstash documentation

for path.config setting:
files are read from the directory in alphabetical order

That means, you need to reconfigure your ls to this:
ls -larh
because manpage says:
https://man7.org/linux/man-pages/man1/ls.1.html
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.

Now, if your filter section is not working, you want to enable following debug settings:
log.level to debug
config.debug to true

Wih these 2 settings in place restarting logstash and looking to the start of the logstash log file, you will see if your filter from file 51-... is even picked up.

Once you have this confirmed, let's review your configuration snippet itself

One thine I have realized is in the grok pattern:

      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

Are you sure, you want at the end of of the first QUOTEDSTRING do a new line like you are doing now, or did you do this on purpose of this configuration discuss entry?

Because it should be normally in one line like this if I am not wrong.

      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

You know your original messages best.

Finally by looking to the GROK pattern itself it looks like a slight variant of a HTTP format.
Are you however sure that there is not closing quote missing right before http_status field like that?
%{NOTSPACE}\" %{INT:http_status}

Finally it might make sense to mention HTTPD_COMMONLOG Grok pattern

So you could change your Grok pattern into something like this:
%{HTTPD_COMMONLOG} (?:-|%{QUOTEDSTRING}) (?:-|%{QUOTEDSTRING}) (?:-|%{BASE16FLOAT:response_time})
However it might result in some other field names, just to make you aware of that.

1 Like

Thanks for giving me time. My filter is working now.
My log format is like below:

10.74.10.184 - - [02/Apr/2021:03:14:14 +0600] "GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1" 200 2951 "-" "BSS-API/2021.12" 117

and my Gork is like below:

%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:resp
onse_time}

for "GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1"

I use below:
\"%{NOTSPACE:request_type} to get http method like GET POST PUT etc
and then
%{NOTSPACE:URIpath} to get /api/v1/subscriptions/26802505/consumption-details
and use %{NOTSPACE} for HTTP/1.1"

I'm checking the HTTPD_COMMONLOG gork patteren.

Made my grok filter in one line. It is working all together.

You spent lot of time for me.... thanks once again.

Glad to hear that it's working for you out.