# Logstash pipeline continue on error?

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787>\
**Category:** Logstash\
**Created:** [November 22, 2017, 7:46pm UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787 "2017-11-22T19:46:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![troy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/troy/32/2547_2.png) [@troy](https://discuss.elastic.co/u/troy)\
**Post date:** [November 22, 2017, 7:46pm UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787/1 "2017-11-22T19:46:29Z")

</div>

We're running a logstash 6.0.0 instance w/ an http\_input plugin. The plugin expects to receive an array of objects. Some bad data made it to the instance last night that was an array w/ an integer in it instead of an object. That caused this error:

> 2017-11-22T05:08:51,063][FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\<NoMethodError: undefined method `empty?' for 1511326405115:Fixnum>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-split-3.1.4/lib/logstash/filters/split.rb:89:in `block in filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-split-3.1.4/lib/logstash/filters/split.rb:88:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi\_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter\_delegator.rb:48:in `multi_filter'", "(eval):428:in `block in filter\_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:501:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:477:in `worker\_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:439:in `block in start\_workers'"]}

The pipeline uses a persistent queue. Logstash quit when it encountered this error and restarted. It then tried to process the event again, which generated the same error and it quit again. Infinite loop time. The cpu spiked and the VM crashed.

I read some discussion about a pipeline.continue\_on\_error setting that was in version 5.x but I don't see it in 6.0? The dead letter queue seems nice but looks like it only works w/ elasticsearch outputs. I'm probably just missing something. Anyway, all this was to ask how can I configure logstash to not quit on error? Ideally I'd like it to log the error and then move on to the next event.

Thanks!

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 23, 2017, 7:13pm UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787/2 "2017-11-23T19:13:17Z")

</div>

In my opinion, if you can't guarantee that the field value you are splitting is an array you should test the event in a conditional before the split filter.  
If `field` normally has an array and its not empty then `[field][0]` should return a "truthy" value and if not it returns a nil which is "falsey".

```auto
  if [field][0] {
    split {
      
    }
  }

```

---

<div class="post-metadata">

**Author:** ![troy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/troy/32/2547_2.png) [@troy](https://discuss.elastic.co/u/troy)\
**Post date:** [November 27, 2017, 11:24pm UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787/3 "2017-11-27T23:24:27Z")

</div>

Thanks Guy. I agree some defensive programming is in order to guard against this but unfortunately bad data sometimes gets missed. I'm hoping there's a mechanism for dealing w/ exceptions in a pipeline that I'm just not seeing? Doesn't seem like an event exception should crash logstash and the VM it's in.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 28, 2017, 7:43am UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787/4 "2017-11-28T07:43:52Z")

</div>

Yes, I agree that exceptions should not halt LS.

An issue that is a very hard problem to solve is, more for plugin authors, deciding whether an exception is soft or hard and what the correct response should be.

We are adding Dead Letter Queue support to plugins, but they have to be individually coded to make the decision to stop LS, a bug that causes every event to fail or a transient event related one that should be written to the DLQ.

Also, we are still deciding on how you the user should come to know that events are piling up in the DLQ, how to inspect them and how to go about building a fix for the DLQ entries. Eventually, with consistent DLQ support across all plugins, the DLQ will have entries with different kinds of faults so the UX for split, repair and ingest is a hard one to crack.

At the moment, we don't have enough metadata on where the event came from and which config it travelled through etc.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 26, 2017, 7:53am UTC](https://discuss.elastic.co/t/logstash-pipeline-continue-on-error/108787/5 "2017-12-26T07:53:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.