# How does String search in Logstash filter work? **URL:** \ **Category:** Logstash\ **Created:** [November 13, 2017, 10:04pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467 "2017-11-13T22:04:10Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 13, 2017, 10:04pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/1 "2017-11-13T22:04:10Z")
Hi, I am using following filter in Logstash, ``` if "INFO" in [message] and "instance" in [message] { mutate{ remove_field => ["name"] } } ``` And it's doing nothing. I don't understand what the problem is. Also, it is actually "instance:" , will I have to provide the exact string instead of "instance"? (I am not including : ) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 6:15am UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/2 "2017-11-14T06:15:13Z")
> And it's doing nothing. I don't understand what the problem is. Whether your configuration works depends on what the events look like, right? > Also, it is actually "instance:" , will I have to provide the exact string instead of "instance"? No, the `in` operator performs a substring search. It's not an equality comparison. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 2:43pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/3 "2017-11-14T14:43:56Z")
Here is the sample input, ``` 2017-11-13 21:30:48:483195 [139718491015040 rid:] INFO [] Server: listens on port 9111 2017-11-13 21:30:48:4858 [139718119044864 rid:] INFO instance: started client connection thread, address of client: xx.xx.xx, port: 11111 2017-11-13 21:30:48:48605 [139718119044864 rid:] INFO [] Server: started client connection thread, address of client: xx.xx.xx, port: 11111 2017-11-13 21:30:48:98756 [139718119044864 rid:] INFO instance: started client connection thread, address of client: xx.xx.xx, port: 11111 2017-11-13 21:30:49:48606 [139718119044864 rid:] INFO [] [Query Timestamp: 1510626648963169] Server: received query: address of client: xx.xx.xx, port: 11111 ``` Here is the config, ``` Input{ file{ path=>"/opt/server.log" start_position=> "beginning" sincedb_path => "/opt/failure_sincedb.log" type=>"failure" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => previous auto_flush_interval => 10 max_lines => 100000 max_bytes => "1000 MiB" } } } filter{ if "INFO" in [message] and "instance:" in [message]{ grok { match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}INFO\s%{GREEDYDATA:error}"} } mutate{ remove_field => ["message"] add_field =>{"component" => "Queries"} } date { match => ["timestamp", "ISO8601"] target => "time" } mutate{ remove_field => ["timestamp"] } } } output{ file{ path => "/tmp/logstash/server10.log" } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 2:46pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/4 "2017-11-14T14:46:12Z")
None of those sample lines contain "instance:". ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 2:47pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/5 "2017-11-14T14:47:14Z")
Sorry, edited the post can you please check once again? 2nd and 4th lines ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 2:56pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/7 "2017-11-14T14:56:35Z")
Okay, so far so good. Please show an event that has been processed by Logstash. Use a `stdout { codec => rubydebug }` output. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 3:09pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/8 "2017-11-14T15:09:01Z")
I am getting \_dateparsefailure, ``` { "path" => "path/test.log", "component" => "Queries", "@timestamp" => 2017-11-14T15:04:29.322Z, "@version" => "1", "host" => "xxx", "type" => "failure", "error" => "instance: started client connection thread, address of clie nt: xx.xx.xx, port: 11111\r", "tags" => [ [0] "_dateparsefailure" ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 3:23pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/9 "2017-11-14T15:23:23Z")
Okay, but then your conditional clearly works. Since you're deleting the `timestamp` field I can't tell what you're asking the date filter to parse. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 3:37pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/10 "2017-11-14T15:37:32Z")
I am deleting timestamp field after applying date filter. It shouldn't have any effect, right? Also I have updated, `match => ["timestamp", "yyyy-MM-dd HH:mm:ss:SSSSSS"]` It's working without any issue. ISO8601 is the culprit here? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 3:56pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/11 "2017-11-14T15:56:06Z")
> I am deleting timestamp field after applying date filter. It shouldn't have any effect, right? Well, it severely affects my ability to debug your problem when you're destroying the evidence. > ISO8601 is the culprit here? Seems so. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 4:06pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/12 "2017-11-14T16:06:22Z")
Here is the output with timestamp field(using match =\> ["timestamp", "yyyy-MM-dd HH:mm:ss:SSSSSS"] ), ``` { "path" => "path/test.log", "component" => "Queries", "@timestamp" => 2017-11-14T16:04:00.621Z, "@version" => "1", "host" => "xxxx", "time" => 2017-11-14T02:30:48.980Z, "type" => "failure", "error" => "instance: started client connection thread, address of clie nt: xx.xx.xx, port: 11111\r", "timestamp" => "2017-11-13 21:30:48:98" } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 7:32pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/13 "2017-11-14T19:32:46Z")
Perhaps you need SS instead of SSSSSS in the date pattern. To deal with microseconds with varying number of digits you might have to use multiple date patterns. You could also truncate the last the timestamp to only include milliseconds since that's the precision ES can store anyway. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [November 14, 2017, 7:47pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/14 "2017-11-14T19:47:07Z")
The config what I gave to you is a part of config file. I have 4 different conditions in the filter and for one of the conditions I have 3 other conditions to be checked. (instance: falls under this condition). I am sending th logs over http to another logstash instance. Will too many filter on Logstash result in loosing data? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2017, 7:47pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467/15 "2017-12-12T19:47:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.