Regexp in conditional


I'm trying to apply a tag based on the contents of a field. I've been trying it like this:

if [field.keyword] =~ /^(TEST|test)-.*$/ {
          mutate { add_tag => [ "TEST" ] } }

The above does not work, the tag never gets applied even though the contents of the field looks like this:


What is the issue with the regex above?

Does your field name have a period in it, or is your field an object with a keyword field inside? If it is the latter then use [field][keyword].

You're right about the object with a keyword field inside it, i've changed it to [field][keyword] but the problem still persists. The tag "TEST" is not being applied even though the contents of the field contains TEST-SOMEDATA.

Well the regex works. If I run

    mutate { add_field => { "[field][keyword]" => "TEST-SOMEDATA" } }
    if [field][keyword] =~ /^(TEST|test)-.*$/ {
        mutate { add_tag => [ "TEST" ] }

then the tag gets added. I see that you have an extra } on the mutate filter. Is that intended?

Hmm, i think there is something wrong with this configuration, but it still runs..

filter {
  if "syslog" in [tags] or "syslog-tcp" in [tags] and "pre-processed" not in [tags] {
    if "test" in [ServiceName] {
        mutate {
            add_tag => [ "test2" ]
        grok {
        patterns_dir => ["/etc/logstash/conf.d/patterns"]
        break_on_match => false
        match => [
            "message", "NetworkDeviceName=%{HOSTNAME:KundID-SiteID}",
            "message", "NOTICE Passed-Authentication: %{CISCO_REASON:Authentication}",
            "message", "NOTICE Failed-Attempt: %{CISCO_REASON:Authentication}",
            "message", "NAS-IP-Address=%{IP:Switch_IP}",
            "message", "Framed-IP-Address=%{IP:Client_IP}",
            "message", "NAS-Port-Id=%{NOTSPACE:Switch_port}",
            "message", "Calling-Station-ID=%{MAC:Client_MAC_address}",
            "message", "Response={User-Name=%{PROG:Authenticator_ID}",
            "message", "Response=\\{User-Name=%{MAC:Authenticator_ID}",
            "message", "Response={UserName=%{MAC:Authenticator_ID}",
            "message", "Response=\\{UserName=%{MAC:Authenticator_ID}",
            "message", "EndPointMACAddress=%{MAC:Client_MAC_address}",
            "message", "NetworkDeviceGroups=Location(?<location>(#.*?)\,)",
            "message", "SelectedAuthorizationProfiles=%{USERNAME:auth_profile}",
            "message", "PostureAssessmentStatus=%{HOSTNAME:posture_status}"
        mutate {
           gsub => [
           # remove all "," on field Switch_port
           "Switch_port", ",", "",
           # remove all ";\" on field Authenticator_ID
           "Authenticator_ID", "[;\\]", ""
        if [KundID-SiteID][keyword] =~ /^(TEST|test)-.*$/ {
          mutate { add_tag => [ "TEST" ] } }

The .keyword field does not exist until the data is ingested into elasticsearch. In logstash you should test

if [KundID-SiteID] =~ /^(TEST|test)-.*$/ {
1 Like

This worked, thank you for the assistance!

I have another question, the GROK fields only shows in the first ELK stack, not the second in which we forward the same events. Is there a way to keep the fields being applied by the first logstash instance in order to prevent performing the same grok operations again?

Hopefully that makes sense :slight_smile:

If you are forwarding the entire event it should retain any fields you are adding using grok. If you just forward the original [message] field then it will not.

How are you forwarding events?

Like this:

syslog        {
                    host => ""
                    port => 514
                    protocol => "tcp"
                    id => "syslog_output_example"

Perhaps i should specify the "message" parameter to something else other than the default %{message} ? Would %{_source} work?

You are using syslog output and input to move events between logstash instances? I would suggest switching to a lumberjack output and a beats input. If you are really married to syslog you can do something like this to send the entire event.