JMS input getting an error JMS Consumer Died

Elastic Stack Logstash
1

Hi all,

I've got a JMS input in my logstash config getting messages from a IBM MQ and i'm getting this error:

JMS Consumer died {:exception=>com.ibm.msg.client.jms.DetailedMessageEOFException: JMSWMQ2002: Failed to get a message from destination 'XXXX'

After digging a bit i find the mqjms.log with this entry:

Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

I've question our infrastructure guys and none of our FW or something like this interrupted the consuming, but we keep getting this error.
The strange thing is that the process runs, but occasionally keeps getting this error.

Anyone passed by this kind of error?

Cheers,

Nuno Ferreira

2

No one passes this kind of error?

Here are more details:

[2019-05-22T11:31:53,591][WARN ][logstash.inputs.jms ] JMS Consumer died {:exception=>com.ibm.msg.client.jms.DetailedMessageEOFException: JMSWMQ2002: Failed to get a message from destination 'Q.LOG.MESSAGE'.
IBM MQ classes for JMS attempted to perform an MQGET; however IBM MQ reported an error.
Use the linked exception to determine the cause of this error., :backtrace=>["com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(com/ibm/msg/client/wmq/common/internal/Reason.java:539)", "com.ibm.msg.client.wmq.common.internal.Reason.createException(com/ibm/msg/client/wmq/common/internal/Reason.java:215)", "com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(com/ibm/msg/client/wmq/internal/WMQMessageConsumer.java:266)", "com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(com/ibm/msg/client/wmq/internal/WMQMessageConsumer.java:156)", "com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(com/ibm/msg/client/wmq/internal/WMQConsumerShadow.java:1848)", "com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(com/ibm/msg/client/wmq/internal/WMQSyncConsumerShadow.java:231)", "com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(com/ibm/msg/client/wmq/internal/WMQConsumerShadow.java:1480)", "com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(com/ibm/msg/client/wmq/internal/WMQMessageConsumer.java:670)", "com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.java:1074)", "com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.java:692)", "com.ibm.mq.jms.MQMessageConsumer.receive(com/ibm/mq/jms/MQMessageConsumer.java:209)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:455)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:316)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.message_consumer.get(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/message_consumer.rb:33)", "org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:117)", "org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.mq_workaround.get(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/mq_workaround.rb:42)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.message_consumer.each(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/message_consumer.rb:69)", "org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:117)", "org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.mq_workaround.each(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/mq_workaround.rb:36)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.session.consume(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/session.rb:402)", "org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:117)", "org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.mq_workaround.consume(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/mq_workaround.rb:6)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_jms_minus_3_dot_0_dot_5_minus_java.lib.logstash.inputs.jms.run_consumer(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jms-3.0.5-java/lib/logstash/inputs/jms.rb:189)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:286)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:266)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.connection.session(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/connection.rb:258)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.connection.session(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/connection.rb:71)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:286)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:266)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.connection.start(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/connection.rb:53)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jruby_minus_jms_minus_1_dot_3_dot_0_minus_java.lib.jms.connection.session(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jruby-jms-1.3.0-java/lib/jms/connection.rb:70)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_jms_minus_3_dot_0_dot_5_minus_java.lib.logstash.inputs.jms.run_consumer(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jms-3.0.5-java/lib/logstash/inputs/jms.rb:186)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_jms_minus_3_dot_0_dot_5_minus_java.lib.logstash.inputs.jms.RUBY$method$run_consumer$0$VARARGS(D_3a_/elk/logstash_minus_6_dot_7_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_jms_minus_3_dot_0_dot_5_minus_java/lib/logstash/inputs/D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jms-3.0.5-java/lib/logstash/inputs/jms.rb)", "D_3a_.elk.logstash_minus_6_dot_7_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_jms_minus_3_dot_0_dot_5_minus_java.lib.logstash.inputs.jms.run(D:/elk/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jms-3.0.5-java/lib/logstash/inputs/jms.rb:257)",

3

hi @bigster

here, you can find connection example to websphere mq:

but currently, input jms plugin haven't opportunity to reconnect:

4

Hi,

I've got that post already, my configuration its OK.
I think this is not a connect issue but a load issue.
I've made some tests and it seems that when i get a real large message in the MQ Manager the JMS input hangs with the error above.
I process a few messages and everything works great until the next large message.
The size of the message seems to be 5MB of size.

Any ideas about this behavior? I'm wrong? How can i check if this is the issue i'm experiencing?

Cheers,

5

@bigster, what about channel throughput?(I don't remember a name of this parameter)

6

Hi don't know this parameter.
Where can i find that parameter to test the behavior?

7

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.explorer.doc/e_properties_channels.htm
Maximum message length of channel
by default, it's equal to 4mb.
So, via Mq explorer or command line you need increase this parameter.

8

Hi,

This was the problem, i was connecting to a channel that has the max size inferior of the queue, and que MQ drops the connection.
Created a new channel with the max with the same size of the max of the queue and everything works.

Cheers.

1 Like
9

Great! You are welcome!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.