Logstash pipeline order

Elastic Stack Logstash
1

Hello All,

Is there any order of logstash pipelines config: I've below pipelines:

[root@gzp-p-qv-logst-3 pipeline]# ls -larth
total 60K
drwxr-xr-x. 3 logstash logstash   42 Oct 11  2019 ..
-rw-r--r--. 1 root     root       99 Nov 11  2019 01-input.conf.j2
-rw-r--r--. 1 root     root      784 Nov 11  2019 50-auditbeat.conf.j2
-rw-r--r--. 1 root     root     1019 Nov 11  2019 50-cassandra.conf.j2
-rw-r--r--. 1 root     root      963 Nov 11  2019 50-httperror.conf.j2
-rw-r--r--. 1 root     root      799 Nov 11  2019 50-kerberos.conf.j2
-rw-r--r--. 1 root     root     4.5K Nov 11  2019 50-platform.conf.j2
-rw-r--r--. 1 root     root     3.6K Nov 11  2019 50-realm.conf.j2
-rw-r--r--. 1 root     root      565 Nov 11  2019 50-secure.conf.j2
-rw-r--r--. 1 root     root      372 Nov 11  2019 98-default_indexes.conf.j2
-rw-r--r--. 1 root     root     2.4K Nov 13  2019 97-outputlogstash.conf
-rw-r--r--. 1 root     root      757 Mar  2 01:41 49-nginx.conf.j2
-rw-r--r--. 1 root     root     2.3K Apr  1 18:35 99-output.conf
-rw-r--r--. 1 root     root      566 Apr  1 19:04 51-applog.conf.j2
drwxr-xr-x. 2 root     root     4.0K Apr  1 21:52 .
[root@gzp-p-qv-logst-3 pipeline]#

I've created 51-applog.conf.j2 like below:

filter {
  if ("qapi" in [tags])
   {
    #mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}
    add_field => [ "tag", "applog" ]
    }

    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }

   mutate { remove_field => [ "host", "message", "@version","timestamp" ] }
  }

}

But seems not working.....
Please help me in understanding the order of pipeline and naming conventions.

Regards.
Banik

2

First of all, looking to the logstash documentation

for path.config setting:
files are read from the directory in alphabetical order

That means, you need to reconfigure your ls to this:
ls -larh
because manpage says:
https://man7.org/linux/man-pages/man1/ls.1.html
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.

Now, if your filter section is not working, you want to enable following debug settings:
log.level to debug
config.debug to true

Wih these 2 settings in place restarting logstash and looking to the start of the logstash log file, you will see if your filter from file 51-... is even picked up.

Once you have this confirmed, let's review your configuration snippet itself

One thine I have realized is in the grok pattern:

      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

Are you sure, you want at the end of of the first QUOTEDSTRING do a new line like you are doing now, or did you do this on purpose of this configuration discuss entry?

Because it should be normally in one line like this if I am not wrong.

      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

You know your original messages best.

Finally by looking to the GROK pattern itself it looks like a slight variant of a HTTP format.
Are you however sure that there is not closing quote missing right before http_status field like that?
%{NOTSPACE}\" %{INT:http_status}

Finally it might make sense to mention HTTPD_COMMONLOG Grok pattern

So you could change your Grok pattern into something like this:
%{HTTPD_COMMONLOG} (?:-|%{QUOTEDSTRING}) (?:-|%{QUOTEDSTRING}) (?:-|%{BASE16FLOAT:response_time})
However it might result in some other field names, just to make you aware of that.

1 Like
3

Thanks for giving me time. My filter is working now.
My log format is like below:

10.74.10.184 - - [02/Apr/2021:03:14:14 +0600] "GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1" 200 2951 "-" "BSS-API/2021.12" 117

and my Gork is like below:

%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:resp
onse_time}

for "GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1"

I use below:
\"%{NOTSPACE:request_type} to get http method like GET POST PUT etc
and then
%{NOTSPACE:URIpath} to get /api/v1/subscriptions/26802505/consumption-details
and use %{NOTSPACE} for HTTP/1.1"

I'm checking the HTTPD_COMMONLOG gork patteren.

Made my grok filter in one line. It is working all together.

You spent lot of time for me.... thanks once again.

4

Glad to hear that it's working for you out.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.