Dynamically build the conditional statement from field value

Elastic Stack Logstash
1

Is it possible to use the value of one field in a json document as an argument to a conditional statement?

f.ex:

{
value1: "test"
value2: "This is a string with the value test included"
}

if value2 =~ /value1/ then {
     string contained "test"
 }

Continuing on the same idea. Would it be possible to have a regex as the field value:

{
value1:"test|example"
value2:"String with the word example included"
}

if value" =~ /value1/ then {
    string contained either "test" or "example"
}

Or is there any other way to dynamically build the conditional when you don't know the condition before the message has been received for parsing?

Haven't looked into the ruby code options.. Could that be used to solve it?
Have played around a bit with it, and not been able to sort it out yet... so if anyone has any suggestions.

2

Yes you can use the field value to check for another string or value. So here when you are referring the value from field you need to access like

instead of

3

Still can't make it work.
Did a simple config: (your forgot to enclose the regexp in //, wont run if not)

input {
  stdin {
  }
}
filter {
  mutate {
    add_field => {"value2" => "lookfor"}
  }
}
filter {
  if [message] =~ /[value2]/ {
    mutate { add_field => { "hit" => true } }
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

This should read from stdin, and set the field hit to true if it finds the lookfor string in the input.
BUT, no matter what input i give, the resulting output will contain the hit:true field:

teststring
{
           "hit" => "true",
    "@timestamp" => 2017-07-24T11:02:46.666Z,
        "value2" => "lookfor",
      "@version" => "1",
          "host" => "XXXXXX",
       "message" => "teststring"
}
lookfor
{
           "hit" => "true",
    "@timestamp" => 2017-07-24T11:02:59.976Z,
        "value2" => "lookfor",
      "@version" => "1",
          "host" => "XXXXXX",
       "message" => "lookfor"
}

Seems to me it checks if there exist a field with the name value2 instead, which ofc always are there.

4

You dont need / here just make condition as :
if [message] =~ [value2] {

Also instead of using 2 filter add everything in one filter as below :

filter {
  mutate {
    add_field => {"value2" => "lookfor"}
  }
 if [message] =~ [value2] {
    mutate { add_field => { "hit" => true } }
  }
}
5

As i said, it won't run without the // :

08:17:52.807 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, \", ', / at line 13, column 19 (byte 133) after filter {\n  if [message] =~ "}

Reason for splitting is just that the first filter is to run on another machine in the pipeline, so i keep them separated for clarity while testing.

And if i just use the filter as you show it behaves in the same way, always adding hit => "true" to the final document printed to stdout:

input {
  stdin {
  }
}
filter {
  mutate {
    add_field => {"value2" => "lookfor"}
  }
  if [message] =~ /[value2]/ {
    mutate { add_field => { "hit" => true } }
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Running this:

08:36:07.748 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9601}
This should not add the hit
{
           "hit" => "true",
    "@timestamp" => 2017-07-25T06:36:23.847Z,
        "value2" => "lookfor",
      "@version" => "1",
          "host" => "host",
       "message" => "This should not add the hit"
}
this should add hit as it contains lookfor
{
           "hit" => "true",
    "@timestamp" => 2017-07-25T06:36:38.114Z,
        "value2" => "lookfor",
      "@version" => "1",
          "host" => "host",
       "message" => "this should add hit as it contains lookfor"
}

Running Logstash 5.5.0-1.

6 
if [message] =~ /[value2]/ {

This definitely won't work but try this instead:

if [message] =~ /%{value2}/ {

If that doesn't work either you may have to use a ruby filter.

7

Didn't get lucky on that one either...
But look on the bright side, this gives me an excuse to start learning ruby as well :smile:
Think it should be possible to patch something together using the code option in the filter.

Thanks for the suggestions.

8

In case others are looking for something along the same lines:

filter {
	mutate {add_field => {"value2" => "lookfor"}}
	ruby {
		code => "if (event.get('message') =~ /#{event.get('value2')}/i)
	          event.set('hit','true'); end" 
	}
}

This seems to do the trick. Only weird thing now is why can't it be a one-liner. If i try:

code => "if (event.get('message') =~ /#{event.get('value2')}/i) event.set('hit','true'); end"

it crashes logstash.. But i might have to blame that on my lacking ruby skills :angry:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.