# Ignore log record if log does not contain specific string "jenkins\_build\_number"

**URL:** https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800
**Category:** Logstash
**Created:** [January 8, 2022, 10:47am UTC](https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800 "2022-01-08T10:47:47Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![chandu.2035](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandu.2035/32/78839_2.png) [@chandu.2035](https://discuss.elastic.co/u/chandu.2035)
#### Post date: [January 8, 2022, 10:47am UTC](https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800/1 "2022-01-08T10:47:47Z")

</div>

Hi there,

I would like to ignore inserting entry in Elasticsearch index if the parsed log pattern does not contain the specific string **jenkins\_build\_number**

Here is my filebeat and logstash configs.

## filebeat.yml

```auto
#======================== Filebeat inputs ==========================
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - jenkins.log
  exclude_files: ['.gz$']
  multiline.pattern: '^[a-zA-Z]+\s[0-9]{1,2},\s[0-9]{4}\s[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s(?:AM|am|PM|pm)'
  multiline.negate: true
  multiline.match: after
  fields:
    type: jenkins-server
  fields_under_root: true
#========================== Outputs ================================
#------------------------- Logstash output -------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["XXXXX"]
  bulk_max_size: 200
#======================== Processors ==============================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
    # - add_cloud_metadata: ~

```

* * *

## logstash - pipeline.conf

```auto

input {
 beats {
        port => "9601"
    }
  }
filter {
  if [type] == "jenkins-server" {
# set all messages from the jenkins log as type 'jenkins' and add the @message field.
          mutate {
              add_field => ["@message_type", "jenkins"]
              add_field => ["@message", "%{message}"]
          }
}
  }
# now that we have possibly-multiline events, we can clean them up.
  filter {
# munge the possibly-multiline messages into a single string
      mutate {
          join => ["@message", "\n"]
      }
# split @message into __date and__ msg, and overwrite the @timestamp value.
      grok {
          match => ["@message", "^(?<__date>%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (AM|PM)) (?<__msg>.+)"]
      }
      date {
          match => ["__date", "MMM dd, YYYY HH:mm:ss a"]
      }
# ...now some patterns to categorize specific event types...
# parse build completion messages, adding the jenkins_* fields and the 'build' tag
      grok {
          match => ["@message", "(?<jenkins_job>\S+) #(?<jenkins_build_number>\d+) (?<__msg>.+): (?<jenkins_build_status>\w+)"]
          tag_on_failure => []
          overwrite => true
          add_tag => ['build']
      }

   # convert build number from string to integer
   mutate {
                convert => ["jenkins_build_number", "integer"]
                }

# remove any empty fields
 ruby {
    code => "event.to_hash.delete_if {|field, value| value == '' }"
 }

# tag messages that come from the git SCM plugin (and associated classes)
      grok {
          match => ["@message", "\.git\."]
          tag_on_failure => []
          add_tag => ['git']
      }
# if we have extracted a short message string, replace @message with it now
      if [__msg] {
          mutate {
              replace => ["@message","%{__msg}"]
          }
      }
# convert @message back into an array of lines
      mutate {
          split => ["@message", "\n"]
      }
  }
# clean-up temporary fields and unwanted tags.
  filter {
      mutate {
          remove_field => [
              "message",
              "__msg",
              "__date",
              "dumps1",
              "plugin_command"
          ]
          remove_tag => [
              "multiline",
              "_grokparsefailure"
          ]
      }
  }
# send it on to the elasticsearch
  output {
    elasticsearch {
                hosts => ["XXXXXXXXXXXXXXXX"]

    # username & password to connect to elaticsearch
                user => "XXX"
                password => "XXX"

                action => "index"
                index => "jenkins-%{+YYYY.MM.dd}"
}

 # use this if you want to verify logs are being sent to elasticsearch or not

        stdout { codec => rubydebug }
  }

```

Output:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0aa5f9014a05855792489bca8b2c94822b21458.png)

The highlighted part in the image has 2 records without jenkins\_build\_number and status, so I would like to remove or ignore before sending it Elasticsearch.

Thanks,

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [January 8, 2022, 2:04pm UTC](https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800/2 "2022-01-08T14:04:05Z")

</div>

You can use a conditional to check if the field does not exist and drop it, just put it after the grok filter where you create the field.

```auto
if ![jenkins_build_number] {
    drop {}
}

```

But since you create it in a `grok` filter and you do not have the field in some message, they probably have the tag `_grokparsefailure`, which you can also use to drop messages.

```auto
if "_grokparsefailure" in [tags] {
    drop {}
}

```

---

<div class="post-metadata">

### Author: ![chandu.2035](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandu.2035/32/78839_2.png) [@chandu.2035](https://discuss.elastic.co/u/chandu.2035)
#### Post date: [January 8, 2022, 4:10pm UTC](https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800/3 "2022-01-08T16:10:51Z")

</div>

Thanks Leandro

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 5, 2022, 4:11pm UTC](https://discuss.elastic.co/t/ignore-log-record-if-log-does-not-contain-specific-string-jenkins-build-number/293800/4 "2022-02-05T16:11:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.