# Logstash filter spec passing, but same message fails in real environment

**URL:** <https://discuss.elastic.co/t/logstash-filter-spec-passing-but-same-message-fails-in-real-environment/111810>\
**Category:** Logstash\
**Created:** [December 14, 2017, 4:19pm UTC](https://discuss.elastic.co/t/logstash-filter-spec-passing-but-same-message-fails-in-real-environment/111810 "2017-12-14T16:19:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![anthony-o](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@anthony-o](https://discuss.elastic.co/u/anthony-o)\
**Post date:** [December 14, 2017, 4:19pm UTC](https://discuss.elastic.co/t/logstash-filter-spec-passing-but-same-message-fails-in-real-environment/111810/1 "2017-12-14T16:19:30Z")

</div>

I'm trying to create a logstash filter to parse SPARQL queries and store them in ES.

I've got very strange results: some messages [tested in rspec](https://github.com/anthony-o/docker-elk_logstash-filter-sparql/blob/master/logstash/logstash-filter-sparql/spec/filters/sparql_spec.rb) tests and correctly passing them are failing in Logstash real runtime environment.

I've created a [GitHub project](https://github.com/anthony-o/docker-elk_logstash-filter-sparql) to reproduce this environment & filter code.

Just clone it, then launch `docker-compose up -d`:

```bash
git clone https://github.com/anthony-o/docker-elk_logstash-filter-sparql.git
cd docker-elk_logstash-filter-sparql
docker-compose up -d

```

`docker-compose` will then build [`logstash-filter-sparql`](https://github.com/anthony-o/docker-elk_logstash-filter-sparql/tree/master/logstash/logstash-filter-sparql), then install it on Logstash and launch all the environment: Filebeat \> Logstash(+logstash-filter-sparql) \> Elasticsearch \> Kibana (with all TCP ports increased by one compared to the defaults; for example, Kibana is started on 5602).

I have a "[logs](https://github.com/anthony-o/docker-elk_logstash-filter-sparql/tree/master/logstash/logstash-filter-sparql/spec/filters/logs)" folder that contains 3 SPARQL queries.  
Those ones correctly pass the [rspecs](https://github.com/anthony-o/docker-elk_logstash-filter-sparql/blob/master/logstash/logstash-filter-sparql/spec/filters/sparql_spec.rb) but in Logstash (after having been [harvested by Filebeat](https://github.com/anthony-o/docker-elk_logstash-filter-sparql/blob/master/filebeat/prospectors.d/default.yml)) they randomly fail to be parsed (in that case there is a field `exception` that I set in the `event` with the Ruby exception). We can see for example `#<NoMethodError: undefined method 'first' for nil:NilClass>` for one or more of the 3 requests.  
Here is the full error (it is not always this one nevertheless):

```
'#<NoMethodError: undefined method `first' for nil:NilClass>' at ["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar/parser11.rb:589:in `block in Parser'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:600:in `block in onFinish'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:118:in `block in eval_with_binding'", "org/jruby/RubyBasicObject.java:1691:in `instance_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:118:in `eval_with_binding'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:599:in `onFinish'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/ebnf-1.1.2/lib/ebnf/ll1/parser.rb:402:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar/parser11.rb:1525:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sparql-2.2.2/lib/sparql/grammar.rb:179:in `parse'", "/usr/share/logstash/vendor/local_gems/761680f2/logstash-filter-sparql-0.1.0/lib/logstash/filters/sparql.rb:44:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:48:in `multi_filter'", "(eval):46:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:455:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:434:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]

```

If all the 3 requests are correctly parsed and displayed on Kibana, you can relaunch everything with `docker-compose down --volumes && docker-compose build && docker-compose up -d`

So I'm wondering: are the rspec execution environment and the Logstash runtime environment really the same?  
How could we explain this behavior?  
In case, I also opened [an issue at `ruby-rdf/sparql`](https://github.com/ruby-rdf/sparql/issues/31).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2018, 4:20pm UTC](https://discuss.elastic.co/t/logstash-filter-spec-passing-but-same-message-fails-in-real-environment/111810/2 "2018-01-11T16:20:16Z")

</div>

Closed on January 11, 2018, 4:20pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.