Regexp in conditional

Elastic Stack Logstash
1

Hi,

I'm trying to apply a tag based on the contents of a field. I've been trying it like this:

if [field.keyword] =~ /^(TEST|test)-.*$/ {
          mutate { add_tag => [ "TEST" ] } }
       }

The above does not work, the tag never gets applied even though the contents of the field looks like this:

TEST-somedata
or
test-somedata

What is the issue with the regex above?

2

Does your field name have a period in it, or is your field an object with a keyword field inside? If it is the latter then use [field][keyword].

3

You're right about the object with a keyword field inside it, i've changed it to [field][keyword] but the problem still persists. The tag "TEST" is not being applied even though the contents of the field contains TEST-SOMEDATA.

4

Well the regex works. If I run

    mutate { add_field => { "[field][keyword]" => "TEST-SOMEDATA" } }
    if [field][keyword] =~ /^(TEST|test)-.*$/ {
        mutate { add_tag => [ "TEST" ] }
    }

then the tag gets added. I see that you have an extra } on the mutate filter. Is that intended?

5

Hmm, i think there is something wrong with this configuration, but it still runs..

filter {
  if "syslog" in [tags] or "syslog-tcp" in [tags] and "pre-processed" not in [tags] {
    if "test" in [ServiceName] {
        mutate {
            add_tag => [ "test2" ]
        }
        grok {
        patterns_dir => ["/etc/logstash/conf.d/patterns"]
        break_on_match => false
        match => [
            "message", "NetworkDeviceName=%{HOSTNAME:KundID-SiteID}",
            "message", "NOTICE Passed-Authentication: %{CISCO_REASON:Authentication}",
            "message", "NOTICE Failed-Attempt: %{CISCO_REASON:Authentication}",
            "message", "NAS-IP-Address=%{IP:Switch_IP}",
            "message", "Framed-IP-Address=%{IP:Client_IP}",
            "message", "NAS-Port-Id=%{NOTSPACE:Switch_port}",
            "message", "Calling-Station-ID=%{MAC:Client_MAC_address}",
            "message", "Response={User-Name=%{PROG:Authenticator_ID}",
            "message", "Response=\\{User-Name=%{MAC:Authenticator_ID}",
            "message", "Response={UserName=%{MAC:Authenticator_ID}",
            "message", "Response=\\{UserName=%{MAC:Authenticator_ID}",
            "message", "EndPointMACAddress=%{MAC:Client_MAC_address}",
            "message", "NetworkDeviceGroups=Location(?<location>(#.*?)\,)",
            "message", "SelectedAuthorizationProfiles=%{USERNAME:auth_profile}",
            "message", "PostureAssessmentStatus=%{HOSTNAME:posture_status}"
        ]
        }
        mutate {
           gsub => [
           # remove all "," on field Switch_port
           "Switch_port", ",", "",
           # remove all ";\" on field Authenticator_ID
           "Authenticator_ID", "[;\\]", ""
                  ]
         }
        if [KundID-SiteID][keyword] =~ /^(TEST|test)-.*$/ {
          mutate { add_tag => [ "TEST" ] } }
       }
    }
  }
6

The .keyword field does not exist until the data is ingested into elasticsearch. In logstash you should test

if [KundID-SiteID] =~ /^(TEST|test)-.*$/ {
1 Like
7

This worked, thank you for the assistance!

I have another question, the GROK fields only shows in the first ELK stack, not the second in which we forward the same events. Is there a way to keep the fields being applied by the first logstash instance in order to prevent performing the same grok operations again?

Hopefully that makes sense :slight_smile:

8

If you are forwarding the entire event it should retain any fields you are adding using grok. If you just forward the original [message] field then it will not.

How are you forwarding events?

9

Like this:

syslog        {
                    host => "host.example.com"
                    port => 514
                    protocol => "tcp"
                    id => "syslog_output_example"
                }

Perhaps i should specify the "message" parameter to something else other than the default %{message} ? Would %{_source} work?

10

You are using syslog output and input to move events between logstash instances? I would suggest switching to a lumberjack output and a beats input. If you are really married to syslog you can do something like this to send the entire event.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.