# NEW to ELK: using conditionals in filter

**URL:** <https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601>\
**Category:** Logstash\
**Created:** [September 9, 2019, 5:21am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601 "2019-09-09T05:21:40Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hAh0L13](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hah0l13/32/53803_2.png) [@hAh0L13](https://discuss.elastic.co/u/hAh0L13)\
**Post date:** [September 9, 2019, 5:21am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/1 "2019-09-09T05:21:40Z")

</div>

Hello. Stuck in configuring pipeline. My filebeat agent send apache\_access logs with "fields.type = apache\_access" option:

> ```
> filebeat.inputs:
> - type: log
> enabled: true
> paths:
> - D:\log\apache\access-*.log
> fields:
> type: apache_access
> 
> ```

My conditions in filter section of Logstash pipeline isn't working. I try

- if [type] == "apache\_access"
- if [fileds.type] == "apache\_access"
- if "apache\_access" in [\_source.fields.type]
- if "apache\_access" in [fields.type]

Without "if" statement, grok filter for HTTPD\_COMMONLOG works perfectly. Without - no error and no filter in Kibana:

> **Kibana JSON**
>
> {  
> "\_index": "xxx-2019.09.09",  
> "\_type": "doc",  
> "\_id": "6apiFG0BnV6XU8Emsk4W",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "message": "10.22.11.10 - - [09/Sep/2019:12:57:53 +0800] "GET /yyy/public/images/logo-xxxx.png HTTP/1.1" 304 -",  
> "@timestamp": "2019-09-09T04:57:54.264Z",  
> "ecs": {  
> "version": "1.0.1"  
> },  
> "host": {  
> "name": "xxxx"  
> },  
> "tags": ,  
> "input": {  
> "type": "log"  
> },  
> "fields": {  
> "type": "apache\_access"  
> },  
> "@version": "1",  
> "agent": {  
> "version": "7.3.1",  
> "ephemeral\_id": "c37a231e-2534-47c7-8883-5eb592b7e844",  
> "hostname": "xxxx",  
> "id": "ea151f77-6c61-487b-b28f-35e0686f58f7",  
> "type": "filebeat"  
> },  
> "log": {  
> "file": {  
> "path": "D:\log\apache\access-2019-09-09.log"  
> },  
> "offset": 57684  
> }  
> },  
> "fields": {  
> "@timestamp": [  
> "2019-09-09T04:57:54.264Z"  
> ]  
> },  
> "sort": [  
> 1568005074264  
> ]  
> }

Whats's wrong with conditional? How can I separate logs - my goal is send several types of logs (apache, mysql, some other software) by one filebeat with different fields.type option and filtering in logstash pipeline with input - beats

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 9, 2019, 5:36am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/2 "2019-09-09T05:36:47Z")

</div>

@hAh0L13, The below pattern should work:

```auto
if [type] == "application_log"

```

You need to define this `if` condition in `output` filter also in logstash.

Thanks.

---

<div class="post-metadata">

**Author:** ![hAh0L13](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hah0l13/32/53803_2.png) [@hAh0L13](https://discuss.elastic.co/u/hAh0L13)\
**Post date:** [September 9, 2019, 5:45am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/3 "2019-09-09T05:45:41Z")

</div>

My logstash config:

> **logstash.conf**
>
> input {  
> beats {  
> port =\> 5044  
> }  
> }
> 
> filter {  
> if [type] == "apache\_access" {  
> grok {  
> match =\> { "message" =\> ["%{HTTPD\_COMMONLOG}", "%{HTTPD\_COMBINEDLOG}"] }  
> }  
> date {  
> match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]  
> }  
> mutate {  
> add\_tag =\> ["apache\_access"]  
> }  
> }  
> if "beats\_input\_codec\_plain\_applied" in [tags] {  
> mutate {  
> remove\_tag =\> ["beats\_input\_codec\_plain\_applied"]  
> }  
> }  
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
> }  
> }

I expect "true" in conditional, then grok "message" + retrieve timestamp with data + adding tag with mutate. And for all cases delete standart tag. Now it only delete standart tag. If i delete IF statement, it works fine.

@Tek_Chand, where is "application\_log" in my message from filebeat? I should replace fields.type in filebeat config from "apache\_access" to "apache\_log"?

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 9, 2019, 5:56am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/4 "2019-09-09T05:56:42Z")

</div>

@hAh0L13,

> [@hAh0L13](#):
>
> where is "application\_log" in my message from filebeat?

`application_log` is just for reference you can change it as per your settings.

In your `logstash` configuration you have used `if` condition but didn't used `else`. But you need to use `else` also.

Your configuration should be look like below:

```auto
input {
  beats {
    port => 5044   
  }
}
filter {
if [type] == "apache_access" {
grok {
match => { "message" => ["%{HTTPD_COMMONLOG}", "%{HTTPD_COMBINEDLOG}"] ] }
}
date {
      date {
match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
add_tag => ["apache_access"]
   }
}
else {
grok {
match => { "message" => ["(?<date-time>[\w\s\d\:]+)\s(?<IP>192.168.50.1)\s(?<port>582)\:\s(?<message>.*)" ] }
}
}
}
output {
  if [type] == "apache_access"
  elasticsearch {
    hosts => ["10.133.58.12:9200"]
    sniffing => true
    manage_template => false
# index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
# index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "application-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
else
  {
elasticsearch {
    hosts => ["10.133.58.12:9200"]
    sniffing => true
    manage_template => false
# index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
}
}
}

```

Above configuration is for your help it may will not work at your end. You may need to make changes at your end as per your detail and configuration like index name, IPs etc.  
Thanks.

---

<div class="post-metadata">

**Author:** ![hAh0L13](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hah0l13/32/53803_2.png) [@hAh0L13](https://discuss.elastic.co/u/hAh0L13)\
**Post date:** [September 9, 2019, 6:27am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/5 "2019-09-09T06:27:29Z")

</div>

Ok. I think found solution - adding "fields\_under\_root: true" option to filebeat let me use "if [type] == "application\_log"" conditional. Maybe using sub-dictionary fields in conditionals is forbidden?

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 9, 2019, 6:31am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/6 "2019-09-09T06:31:42Z")

</div>

@hAh0L13,

> [@hAh0L13](#):
>
> fields\_under\_root: true

Yes, you need to set above setting in your `filebeat.yml`.

Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 9, 2019, 10:52am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/7 "2019-09-09T10:52:01Z")

</div>

> [@hAh0L13](#):
>
> Maybe using sub-dictionary fields in conditionals is forbidden?

No, it is not forbidden, but the syntax to reference a sub-field in logstash is [fields][type]. [fields.type] would reference a field that contains a period in its name.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 7, 2019, 11:04am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601/8 "2019-10-07T11:04:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.