Logstash heartbeat input error

[mostafaelsayed]

Hello All,

I am using heartbeat input to perform a specific job periodically.
However, on my local windows machine, I occasionally keep getting this error message

[2023-02-09T02:35:37,209][ERROR][logstash.javapipeline    ][heartbeat-output][ac8794b92125c73f2c6f5778fd0933f59365fda686df140bdea0f70909cdb874] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:heartbeat-output
  Plugin: <LogStash::Inputs::Heartbeat interval=>1, id=>"ac8794b92125c73f2c6f5778fd0933f59365fda686df140bdea0f70909cdb874", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_f91008ea-a123-4946-b5c6-23794bc5d410", enable_metric=>true, charset=>"UTF-8">, threads=>1, message=>"ok", count=>-1>
  Error: Tried to write to a closed queue.
  Exception: Java::JavaLang::IllegalStateException

This error happens in random times while the Logstash server is up, so I don't know exactly when or why it happens.

However, when trying to run the same configuration on our remote server in a Linux container, it seems to be working without any errors (at least until now and I am still monitoring it).

so, is this a problem with windows only?
if this is the case, is there any workaround for it at least to be able to work locally without having to restart the server every time it happens?

Note: I am using Logstash 7.9.2

Thanks

[rugenl (Len Rugen)]

Does data flow at (most) times through this pipeline? If so, this could be an intermittent problem. I had something vaguely similar on logstash boxes that were WAN away from my elastic cluster. I had persistent queues so data would buffer during brief outages. I ignored it for years

[mostafaelsayed]

Hello @rugenl

The problem is that even when no actual processing happening and no brief shutdown, I keep getting this error.

Also, I can't ignore it if it happened on production as well because it's blocking the pipeline.

The processing of the heartbeat is actually happening when another http input in the pipeline triggered, so, even when no http input event received (system is basically idle, heartbeat does nothing, no shutdowns occurred), I still get this error.

I guess I will have to use the memory queue locally then and see if it will solve the issue.