# LS2 'stuck'?

**URL:** <https://discuss.elastic.co/t/ls2-stuck/35270>\
**Category:** Logstash\
**Created:** [November 23, 2015, 6:20am UTC](https://discuss.elastic.co/t/ls2-stuck/35270 "2015-11-23T06:20:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 23, 2015, 6:20am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/1 "2015-11-23T06:20:51Z")

</div>

Hi group,

I have a LS2 setup taking input from Redis and sending output to remote ES2.

LS2 has multiple multiline filters and two outputs: debug and ES2. LS2 seems to be 'stuck'/'frozen' after going through several entries. Any way to figure out whats going on? Is this a known issue?

If I kill and restart, LS2 comes up fine, goes through several messages from Redis and reaches the same state.

Any thoughts or help?

Thanks!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2015, 6:25am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/2 "2015-11-23T06:25:33Z")

</div>

What does your LS config look like? What is the state of the ES cluster?

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 23, 2015, 6:31am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/3 "2015-11-23T06:31:44Z")

</div>

Thanks for your reply!

ES2 state looks like this: (just one node currently since I am trying things out before going full blown)

```
{
  "cluster_name" : "live",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

```

LS2 config is broken up into multiple files to merge filters. The input and output section looks like this:  
input {  
redis {  
host =\> 'localhost'  
data\_type =\> 'list'  
key =\> 'logstash'  
type =\> 'redis-input'  
}  
}

```
output {
 elasticsearch {
 hosts => ["10.86.205.62:9200"]
}
 stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 23, 2015, 6:50am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/4 "2015-11-23T06:50:48Z")

</div>

Are there any clues in the Logstash and/or Elasticsearch logs? In the Logstash case, cranking up the logging level with `--verbose` or even `--debug` can provide additional clues.

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 23, 2015, 5:23pm UTC](https://discuss.elastic.co/t/ls2-stuck/35270/5 "2015-11-23T17:23:59Z")

</div>

Thanks for your reply!

So, I did start LS2 with --verbose and I see the logs stop at:

```
{:timestamp=>"2015-11-23T01:59:56.058000-0800", :message=>"Using mapping template", :template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"location"=>{"type"=>"geo_point"}}}}}}}, :level=>:info}
{:timestamp=>"2015-11-23T01:59:56.203000-0800", :message=>"New Elasticsearch output", :hosts=>["10.86.205.62:9200"], :level=>:info}

```

Nothing happens here even when LS2 stops sending. I removed the rubydebug stdout just to check whether putting everything on stdout has any effect. But still the same happens.

At the ES2 end, I am capturing packets to see whats coming from LS2 and I see packets stop arriving on ES2. The ES2 logs dont say much except updating indices:

```
[2015-11-23 09:18:37,854][INFO][cluster.metadata] [srv-9] [logstash-2015.10.29] update_mapping [lumberjack]
[2015-11-23 09:19:47,124][INFO][cluster.metadata] [srv-9] [logstash-2015.11.07] update_mapping [lumberjack]

```

Any specific debugs that can be enabled for this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 24, 2015, 6:44am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/6 "2015-11-24T06:44:12Z")

</div>

Reduce the complexity of your system until it starts to work again by e.g. disabling the elasticsearch output and removing filters and whatever else you have. You mention that you're using multline filters. That would be my prime suspect. Logstash will get stuck waiting for the line that should "close" an event if that line never shows up (e.g. because the multiline regexp is wrong).

---

<div class="post-metadata">

**Author:** ![mad\_min](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_min/32/5964_2.png) [@mad\_min](https://discuss.elastic.co/u/mad_min)\
**Post date:** [November 24, 2015, 1:59pm UTC](https://discuss.elastic.co/t/ls2-stuck/35270/7 "2015-11-24T13:59:35Z")

</div>

Thanks @magnusbaeck.

Yeah, thats what I am suspecting as well. Trying to move away from multiline as much as possible. Instinctively, I feel thats been the root-cause of all the issues LS2 is giving me.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:21am UTC](https://discuss.elastic.co/t/ls2-stuck/35270/8 "2017-07-06T05:21:28Z")

</div>