Getting logs from a TOOK having >15ms using logstash?

Elastic Stack Logstash
1

HI,

LOGS:

[2017-01-14 10:48:06,848][WARN ][index.search.slowlog.query] [yaswanth] [bank][0] took[27.8ms], took_millis[27], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[], extra_source[], 
[2017-01-14 10:48:06,851][WARN ][index.search.slowlog.query] [yaswanth] [bank][3] took[33.7ms], took_millis[33], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[], extra_source[],

I am trying to get the logs whose TOOK field is >15ms . I used the below config

input {
  file {
    path => "C:\Users\571952\Desktop\pica.txt"
    start_position => "beginning"
  }
}

filter {
   grok {
        match => [ "message", 

"\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:QUERY}\]%{SPACE}\[%{DATA:QUE

RY1}\]%{SPACE}\[%{DATA:INDEX-NAME}\]\[%{DATA:SHARD}\]%{SPACE}took\[%{DATA:TOOK}\],%{SPACE}

took_millis\[%{DATA:TOOKM}\], types\[%{DATA:types}\], stats\[%{DATA:stats}\], search_type

\[%{DATA:search_type}\], total_shards\[%{NUMBER:total_shards}\], source\[%{DATA:source_query}\], 

extra_source\[%{DATA:extra_source}\],"]
   }
 if [TOOK] > 15ms {
    mutate {
      add_tag => "slowresponse"
    }
  } else {
    drop { }
  }
}
output {
stdout { codec => rubydebug }

}

I am getting error like this:

←[31mfetched an invalid config {:config=>"input {\n  file {\n    path => \"C:\\U
sers\\571952\\Desktop\\pica.txt\"\n    start_position => \"beginning\"\n  }\n}\n
\nfilter {\n   grok {\n        match => [ \"message\", \"\\[%{TIMESTAMP_ISO8601:
TIMESTAMP}\\]\\[%{LOGLEVEL:LEVEL}%{SPACE}\\]\\[%{DATA:QUERY}\\]%{SPACE}\\[%{DATA
:QUERY1}\\]%{SPACE}\\[%{DATA:INDEX-NAME}\\]\\[%{DATA:SHARD}\\]%{SPACE}took\\[%{D
ATA:TOOK}\\],%{SPACE}took_millis\\[%{DATA:TOOKM}\\], types\\[%{DATA:types}\\], s
tats\\[%{DATA:stats}\\], search_type\\[%{DATA:search_type}\\], total_shards\\[%{

Even i tried with removing ms

Thanks

2

I can't spot anything obviously wrong. Comment out pieces of your configuration to narrow things down.

3

If i remove this part it is working fine. Is there any thing wrong in the above code?

Thanks

4

Try:

if [TOOK] > "15ms" {

But! Keep in mind that this'll be a string comparison. If TOOK is "2ms" the expression above will be true. You should strip the "ms" suffix from TOOK and convert it to an integer.

5

Thanks @magnusbaeck

Still it is throwing the error

e[31mException in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<NoMethodError: undefined method `>' for nil:NilClass>, "backtrace"=>["(eval):123:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):121:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):91:in `filter_func'", "F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:265:in `filter_batch'", "F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}e[0m
NoMethodError: undefined method `>' for nil:NilClass
     initialize at (eval):123
           each at org/jruby/RubyArray.java:1613
     initialize at (eval):121
           call at org/jruby/RubyProc.java:281
    filter_func at (eval):91
   filter_batch at F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:267
           each at org/jruby/RubyArray.java:1613
         inject at org/jruby/RubyEnumerable.java:852
   filter_batch at F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:265
    worker_loop at F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:223
  start_workers at F:/logstash-2.4.0/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201

The matching grok pattern has output like this which means working fine

message" => "[2017-01-14 10:48:06,848][WARN ][index.search.slowlog.query] [yaswanth] [bank][0] took[27.8ms], took_millis[27], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[], extra_source[], \r",
        "@version" => "1",
      "@timestamp" => "2017-05-09T16:43:58.995Z",
            "path" => "F:\\logstash-2.4.0\\logstash-2.4.0\\pica.txt",
            "host" => "yaswanth",
       "TIMESTAMP" => "2017-01-14 10:48:06,848",
           "LEVEL" => "WARN",
           "QUERY" => "index.search.slowlog.query",
          "QUERY1" => "yaswanth",
      "INDEX-NAME" => "bank",
           "SHARD" => "0",
            "TOOK" => "27.8ms",
           "TOOKM" => "27",
     "search_type" => "QUERY_THEN_FETCH",
    "total_shards" => "5"

Thanks

6

Are you sure all events have a TOOK field? If you rewrite your conditional to

if [TOOK] and [TOOK] > "15ms" {

Logstash will only attempt the comparison if the TOOK field is set.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.