# Logstash pipeline order

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-order/269057>\
**Category:** Logstash\
**Created:** [April 1, 2021, 4:32pm UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057 "2021-04-01T16:32:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Narayan\_Banik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_banik/32/84627_2.png) [@Narayan\_Banik](https://discuss.elastic.co/u/Narayan_Banik)\
**Post date:** [April 1, 2021, 4:32pm UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057/1 "2021-04-01T16:32:51Z")

</div>

Hello All,

Is there any order of logstash pipelines config: I've below pipelines:

```auto
[root@gzp-p-qv-logst-3 pipeline]# ls -larth
total 60K
drwxr-xr-x. 3 logstash logstash 42 Oct 11 2019 ..
-rw-r--r--. 1 root root 99 Nov 11 2019 01-input.conf.j2
-rw-r--r--. 1 root root 784 Nov 11 2019 50-auditbeat.conf.j2
-rw-r--r--. 1 root root 1019 Nov 11 2019 50-cassandra.conf.j2
-rw-r--r--. 1 root root 963 Nov 11 2019 50-httperror.conf.j2
-rw-r--r--. 1 root root 799 Nov 11 2019 50-kerberos.conf.j2
-rw-r--r--. 1 root root 4.5K Nov 11 2019 50-platform.conf.j2
-rw-r--r--. 1 root root 3.6K Nov 11 2019 50-realm.conf.j2
-rw-r--r--. 1 root root 565 Nov 11 2019 50-secure.conf.j2
-rw-r--r--. 1 root root 372 Nov 11 2019 98-default_indexes.conf.j2
-rw-r--r--. 1 root root 2.4K Nov 13 2019 97-outputlogstash.conf
-rw-r--r--. 1 root root 757 Mar 2 01:41 49-nginx.conf.j2
-rw-r--r--. 1 root root 2.3K Apr 1 18:35 99-output.conf
-rw-r--r--. 1 root root 566 Apr 1 19:04 51-applog.conf.j2
drwxr-xr-x. 2 root root 4.0K Apr 1 21:52 .
[root@gzp-p-qv-logst-3 pipeline]# 

```

I've created 51-applog.conf.j2 like below:

```auto
filter {
  if ("qapi" in [tags])
   {
    #mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}
    add_field => ["tag", "applog"]
    }

    date {
      match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
    }

   mutate { remove_field => ["host", "message", "@version","timestamp"] }
  }

}

```

But seems not working.....  
Please help me in understanding the order of pipeline and naming conventions.

Regards.  
Banik

---

<div class="post-metadata">

**Author:** ![Andre\_Letterer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_letterer/32/42248_2.png) [@Andre\_Letterer](https://discuss.elastic.co/u/Andre_Letterer)\
**Post date:** [April 1, 2021, 11:20pm UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057/2 "2021-04-01T23:20:20Z")

</div>

First of all, looking to the logstash documentation

> **[logstash.yml | Logstash Reference \[7.12\] | Elastic](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html)**

for `path.config` setting:  
`files are read from the directory in alphabetical order`

That means, you need to reconfigure your ls to this:  
`ls -larh`  
because manpage says:  
[https://man7.org/linux/man-pages/man1/ls.1.html](https://man7.org/linux/man-pages/man1/ls.1.html)  
`Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.`

Now, if your filter section is not working, you want to enable following debug settings:  
`log.level` to `debug`  
`config.debug` to `true`

Wih these 2 settings in place restarting logstash and looking to the start of the logstash log file, you will see if your filter from file `51-...` is even picked up.

Once you have this confirmed, let's review your configuration snippet itself

One thine I have realized is in the grok pattern:

```auto
      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING}
 %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

```

Are you sure, you want at the end of of the first QUOTEDSTRING do a new line like you are doing now, or did you do this on purpose of this configuration discuss entry?

Because it should be normally in one line like this if I am not wrong.

```auto
      match => { "message" =>
    ["%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:response_time}"]}

```

You know your original messages best.

Finally by looking to the GROK pattern itself it looks like a slight variant of a HTTP format.  
Are you however sure that there is not closing quote missing right before http\_status field like that?  
`%{NOTSPACE}\" %{INT:http_status}`

Finally it might make sense to mention `HTTPD_COMMONLOG` Grok pattern

> <https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/legacy/httpd#L5>

So you could change your Grok pattern into something like this:  
`%{HTTPD_COMMONLOG} (?:-|%{QUOTEDSTRING}) (?:-|%{QUOTEDSTRING}) (?:-|%{BASE16FLOAT:response_time})`  
However it might result in some other field names, just to make you aware of that.

---

<div class="post-metadata">

**Author:** ![Narayan\_Banik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_banik/32/84627_2.png) [@Narayan\_Banik](https://discuss.elastic.co/u/Narayan_Banik)\
**Post date:** [April 2, 2021, 7:39am UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057/3 "2021-04-02T07:39:27Z")

</div>

> [@Andre\_Letterer](#):
>
> QUOTEDSTRING

Thanks for giving me time. My filter is working now.  
My log format is like below:

```auto
10.74.10.184 - - [02/Apr/2021:03:14:14 +0600] "GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1" 200 2951 "-" "BSS-API/2021.12" 117

```

and my Gork is like below:

```auto
%{IP:client} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\] \"%{NOTSPACE:request_type} %{NOTSPACE:URIpath} %{NOTSPACE} %{INT:http_status} %{NUMBER:payload} %{QUOTEDSTRING} %{QUOTEDSTRING} %{BASE16FLOAT:resp
onse_time}

```

for `"GET /api/v1/subscriptions/26802505/consumption-details HTTP/1.1"`

I use below:  
`\"%{NOTSPACE:request_type}` to get http method like `GET POST PUT` etc  
and then  
`%{NOTSPACE:URIpath}` to get `/api/v1/subscriptions/26802505/consumption-details`  
and use `%{NOTSPACE}` for `HTTP/1.1"`

I'm checking the `HTTPD_COMMONLOG` gork patteren.

Made my grok filter in one line. It is working all together.

You spent lot of time for me.... thanks once again.

---

<div class="post-metadata">

**Author:** ![Andre\_Letterer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_letterer/32/42248_2.png) [@Andre\_Letterer](https://discuss.elastic.co/u/Andre_Letterer)\
**Post date:** [April 2, 2021, 8:34am UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057/4 "2021-04-02T08:34:54Z")

</div>

Glad to hear that it's working for you out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2021, 8:35am UTC](https://discuss.elastic.co/t/logstash-pipeline-order/269057/5 "2021-04-30T08:35:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.