Logstash filter spec passing, but same message fails in real environment

I'm trying to create a logstash filter to parse SPARQL queries and store them in ES.

I've got very strange results: some messages tested in rspec tests and correctly passing them are failing in Logstash real runtime environment.

I've created a GitHub project to reproduce this environment & filter code.

Just clone it, then launch docker-compose up -d:

git clone https://github.com/anthony-o/docker-elk_logstash-filter-sparql.git
cd docker-elk_logstash-filter-sparql
docker-compose up -d

docker-compose will then build logstash-filter-sparql, then install it on Logstash and launch all the environment: Filebeat > Logstash(+logstash-filter-sparql) > Elasticsearch > Kibana (with all TCP ports increased by one compared to the defaults; for example, Kibana is started on 5602).

I have a "logs" folder that contains 3 SPARQL queries.
Those ones correctly pass the rspecs but in Logstash (after having been harvested by Filebeat) they randomly fail to be parsed (in that case there is a field exception that I set in the event with the Ruby exception). We can see for example #<NoMethodError: undefined method 'first' for nil:NilClass> for one or more of the 3 requests.
Here is the full error (it is not always this one nevertheless):

'#<NoMethodError: undefined method `first' for nil:NilClass>' at ["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar/parser11.rb:589:in `block in Parser'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:600:in `block in onFinish'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:118:in `block in eval_with_binding'", "org/jruby/RubyBasicObject.java:1691:in `instance_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:118:in `eval_with_binding'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:599:in `onFinish'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:402:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar/parser11.rb:1525:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar.rb:179:in `parse'", "/usr/share/logstash/vendor/local_gems/761680f2/logstash-filter-sparql-0.1.0/lib/logstash/filters/sparql.rb:44:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:48:in `multi_filter'", "(eval):46:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:455:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:434:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]

If all the 3 requests are correctly parsed and displayed on Kibana, you can relaunch everything with docker-compose down --volumes && docker-compose build && docker-compose up -d

So I'm wondering: are the rspec execution environment and the Logstash runtime environment really the same?
How could we explain this behavior?
In case, I also opened an issue at ruby-rdf/sparql.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.