What is wrong with this filter?

Elastic Stack Logstash
1

Hi,

I have the following filter that passes the configuration test:

./logstash -f ../config/pipeline.conf  --config.test_and_exit
Sending Logstash's logs to /usr/local/rex/logstash-5.0.2/logs which is now configured via log4j2.properties
Configuration OK
[2016-12-29T17:24:28,962][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

    input {
        beats {
            port => "5043"
        }
    }
    filter {
        kv {
        }
     grok { 
        match => { "message" => "%{TIMESTAMP_ISO8601:time}" } 
      }
      mutate {convert => {"TotalTime" => "integer"}}
      if [TotalTime] >  300 { mutate {add_field => ["Over300", 1]}} 
      else if [Totaltime] > 1500 { mutate {add_field => ["Over1500", 1]}}
      else if [TotalTime] > 1800 { mutate {add_field => ["Over1800", 1]}}
      else if [TotalTime] > 2000 { mutate {add_field => ["Over2000", 1]}}
      else if [TotalTime] > 4500 { mutate {add_field => ["Over4500", 1]}}
}
    output {
    stdout { codec => rubydebug }
    	elasticsearch {
    		hosts => ["10.10.1.123:9200","10.10.1.124:9200"]
    		index => ["ngbv1"]
    	}  
  }

But when the pipeline is ran it fails with the following error:

./logstash -f ../config/pipeline.conf  --config.reload.automat
Sending Logstash's logs to /usr/local/rex/logstash-5.0.2/logs which is now configured via log4j2.properties
[2016-12-29T17:34:37,990][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2016-12-29T17:34:38,076][INFO ][org.logstash.beats.Server] Starting server on port: 5043

[2016-12-29T17:34:38,800][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["10.252.173.47:9200", "10.252.26.133:9200", "10.252.26.132:9200", "10.252.115.175:9200", "10.252.113.97:9200"]}
[2016-12-29T17:34:38,931][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-12-29T17:34:38,950][INFO ][logstash.pipeline        ] Pipeline main started
[2016-12-29T17:34:39,000][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[2016-12-29T17:35:17,025][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<NoMethodError: undefined method `>' for nil:NilClass>, "backtrace"=>["(eval):377:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):372:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):254:in `filter_func'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:260:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:186:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:185:in `each'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:258:in `filter_batch'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:246:in `worker_loop'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:225:in `start_workers'"]}
[2016-12-29T17:35:17,231][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<NoMethodError: undefined method `>' for nil:NilClass>, "backtrace"=>["(eval):377:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):372:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):254:in `filter_func'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:260:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:186:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:185:in `each'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:258:in `filter_batch'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:246:in `worker_loop'", "/usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:225:in `start_workers'"]}
NoMethodError: undefined method `>' for nil:NilClass
     initialize at (eval):377
           each at org/jruby/RubyArray.java:1613
     initialize at (eval):372
           call at org/jruby/RubyProc.java:281
    filter_func at (eval):254
   filter_batch at /usr/local/rex/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:260
           call at org/jruby/RubyProc.java:281
           each at /usr/local/rex/logstash-5.0.2/logstashcore/lib/logstash/util/wrapped_synchronous_queue.rb:186
           each at org/jruby/RubyHash.java:1342
2

Please use the </> button to format your code and logs, it makes it much easier to read :slight_smile:

Where is the rest of your config? It mentions beats in the logs but it's not in what you have provided.

3

Thanks. Updated as requested.

4

What does your data look like?

5

This is an event instance:

2016-12-30 13:59:54,743 INFO Component="HO" Version="2.F.T" Application="5.0|HO" EntryCount="318" Returned="12" StatusCode="200" SessionId="5608303524247454937" DiscoveryId="292-172.24.232.155-11914984-1483106394730356" Query="http://blah.com:10124/abc/dfg/p?schema=5.0&node.id=4967753593854790170&client=hello&showings=1&filters=%28catalog.type%3Avod+AND+tag.id.company%3A657829ew419676893105+AND+tag.type%3Amovie+AND+%28tag.encoding%3Ampeg_qam+OR+tag.encoding%3Am3u+OR+tag.encoding%3Ahtml%29%29&fields=program.programId%2Cprogram.title%2Cprogram.ratings%2Cprogram.synopsis%2Cprogram.originalAirDate%2Cprogram.duration%2Cprogram.programType%2Cprogram.starRating%2Cprogram.language%2Cprogram.year%2Cprogram.tags%2Cprogram.sportsSubtitle%2Cprogram.audience%2Cprogram.seasons%2Cprogram.companyIds%2Cprogram.partNumber%2Cprogram.totalParts%2Cprogram.facets%2Cprogram.offers%2Cprogram.images%2Cprogram.seriesId%2Cprogram.episodeTitle%2Cprogram.episodeNumber%2Cprogram.tvSeasonNumber&items.paging=1-12&filter.image.gid=0%2C0&menu.id=6502259236126042170" Client="XRE:X2" PagingStartIndex="1" StatusMessage="OK" TotalTime="13" timer_Solr="5" timer_ShowingJoinQParserPlugin_totalLatency="3" timer_ResponseWriter_Total="6" timer_RequestHandler_createRequestContext="1"

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.