How does String search in Logstash filter work?

Elastic Stack Logstash
1

Hi,

I am using following filter in Logstash,

 if "INFO" in [message] and "instance" in [message] {
      mutate{
                                 remove_field => ["name"]
                        }
}

And it's doing nothing. I don't understand what the problem is.

Also, it is actually "instance:" , will I have to provide the exact string instead of "instance"? (I am not including : )

2

And it's doing nothing. I don't understand what the problem is.

Whether your configuration works depends on what the events look like, right?

Also, it is actually "instance:" , will I have to provide the exact string instead of "instance"?

No, the in operator performs a substring search. It's not an equality comparison.

3

Here is the sample input,

2017-11-13 21:30:48:483195 [139718491015040 rid:            ]  INFO  []  Server: listens on port 9111
    2017-11-13 21:30:48:4858 [139718119044864 rid:            ]  INFO instance: started client connection thread, address of client: xx.xx.xx, port: 11111
    2017-11-13 21:30:48:48605 [139718119044864 rid:            ]  INFO  []  Server: started client connection thread, address of client: xx.xx.xx, port: 11111
    2017-11-13 21:30:48:98756 [139718119044864 rid:            ]  INFO instance: started client connection thread, address of client: xx.xx.xx, port: 11111
    2017-11-13 21:30:49:48606 [139718119044864 rid:            ]  INFO  [] [Query Timestamp: 1510626648963169]  Server: received query: address of client: xx.xx.xx, port: 11111

Here is the config,

Input{
	file{
                path=>"/opt/server.log"
                start_position=> "beginning"
                sincedb_path => "/opt/failure_sincedb.log"
                type=>"failure"

                codec => multiline {
                        pattern => "^%{TIMESTAMP_ISO8601}"
                        negate => true
                        what => previous
                        auto_flush_interval => 10
                        max_lines => 100000
                        max_bytes => "1000 MiB"
                }
        }
}

filter{
	if "INFO" in [message] and "instance:" in [message]{

        grok {
             match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}INFO\s%{GREEDYDATA:error}"}
            }

        mutate{
            remove_field => ["message"]
                add_field =>{"component" => "Queries"}                    
            }

        date {
            match => ["timestamp", "ISO8601"]
            target => "time"
        }

        mutate{
			remove_field => ["timestamp"]
		}
    }

}

output{
 file{
	path => "/tmp/logstash/server10.log"
 }
}
4

None of those sample lines contain "instance:".

5

Sorry, edited the post can you please check once again? 2nd and 4th lines

7

Okay, so far so good. Please show an event that has been processed by Logstash. Use a stdout { codec => rubydebug } output.

8

I am getting _dateparsefailure,

{
          "path" => "path/test.log",
     "component" => "Queries",
    "@timestamp" => 2017-11-14T15:04:29.322Z,
      "@version" => "1",
          "host" => "xxx",
          "type" => "failure",
         "error" => "instance: started client connection thread, address of clie
nt: xx.xx.xx, port: 11111\r",
          "tags" => [
        [0] "_dateparsefailure"
    ]
}
9

Okay, but then your conditional clearly works. Since you're deleting the timestamp field I can't tell what you're asking the date filter to parse.

10

I am deleting timestamp field after applying date filter. It shouldn't have any effect, right?

Also I have updated,

match => ["timestamp", "yyyy-MM-dd HH:mm:ss:SSSSSS"]

It's working without any issue. ISO8601 is the culprit here?

11

I am deleting timestamp field after applying date filter. It shouldn't have any effect, right?

Well, it severely affects my ability to debug your problem when you're destroying the evidence.

ISO8601 is the culprit here?

Seems so.

12

Here is the output with timestamp field(using match => ["timestamp", "yyyy-MM-dd HH:mm:ss:SSSSSS"] ),

{
          "path" => "path/test.log",
     "component" => "Queries",
    "@timestamp" => 2017-11-14T16:04:00.621Z,
      "@version" => "1",
          "host" => "xxxx",
          "time" => 2017-11-14T02:30:48.980Z,
          "type" => "failure",
         "error" => "instance: started client connection thread, address of clie
nt: xx.xx.xx, port: 11111\r",
     "timestamp" => "2017-11-13 21:30:48:98"
}
13

Perhaps you need SS instead of SSSSSS in the date pattern. To deal with microseconds with varying number of digits you might have to use multiple date patterns. You could also truncate the last the timestamp to only include milliseconds since that's the precision ES can store anyway.

14

The config what I gave to you is a part of config file. I have 4 different conditions in the filter and for one of the conditions I have 3 other conditions to be checked. (instance: falls under this condition). I am sending th logs over http to another logstash instance.

Will too many filter on Logstash result in loosing data?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.