Logstash-input-jms problem reading messages from IBM WebSphere MQ (missing valuefier)

alexander-milovidov · April 11, 2017, 2:00pm

I have problem getting messages from IBM WebSphere MQ. It seems that some field(s) in MQMD header has type which is not handled properly. Some messages are read without exception.
When I turn off include_header setting in jms input section, all messages are handled without any errors.
Exception is: java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=java.lang.Byte, simple name=Byte

Logstash version: 5.3.0
Plugin version: logstash-input-jms-3.0.1-java
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
OS: Windows 7 Ent SP1, Linux RedHat (uname -r output: 2.6.32-642.4.2.el6.x86_64)
WebSphere MQ versions 7.5.0.6, 8.0.0.6

Config input section:

input {
    jms {
        yaml_file => "C:/logstash-5.3.0/config/jms.yml"
        yaml_section => "wmq"
        include_header => true
        include_properties => true
        include_body => true
        use_jms_timestamp => false
        destination => "TEST.LOG"
        interval => 3
        runner => async
        add_field => [ "host", "computer_name" ]
    }
}

jms.yml:
---
# IBM WebSphere MQ
wmq:
  :factory: com.ibm.mq.jms.MQQueueConnectionFactory
  :queue_manager: QMNAME
  :host_name: server_name.domain_name
  :channel: LOGSTASH.CHL
  :port: 1414
  # Transport Type: com.ibm.mq.jms.JMSC::MQJMS_TP_CLIENT_MQ_TCPIP
  :transport_type: 1
  :username: test
  :require_jars:
    - C:/IBM/MQ/java/lib/com.ibm.mqjms.jar

I can provide additional details if it is needed.

How to reproduce.
I can attach two sample messages which can be loaded to MQ queue using RFHUTIL (SupportPac IH03) (use "Load Q" button with default settings). One of these messages is processed without error, another message causes exception.

alexander-milovidov · April 11, 2017, 2:02pm

16:16:47.499 [DispatchThread: com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D5143514D4D534B5630323120202058D31E1625C9467C]]] ERROR logstash.inputs.jms - Failed to create event {:message=>Java::ComIbmJms::JMSBytesMessage:Attributes: {:jms_correlation_id=>nil, :jms_delivery_mode_sym=>:persistent, :jms_destination=>nil, :jms_expiration=>0, :jms_message_id=>"ID:414d5120514d4d534b5630323120202058d31e1625c89c43", :jms_priority=>4, :jms_redelivered=>false, :jms_reply_to=>{"CCSID"=>1208, "brokerCCDurSubQueue"=>"SYSTEM.JMS.D.CC.SUBSCRIBER.QUEUE", "JMS_IBM_Unmappable_Action"=>"REPORT", "failIfQuiesce"=>1, "destDescription"=>nil, "readAheadClosePolicy"=>2, "JMS_IBM_Unmappable_Replacement"=>63, "mdWriteEnabled"=>false, "XMSC_WMQ_QUEUE_MANAGER"=>"QMNAME", "timeToLive"=>-2, "putAsyncAllowed"=>-1, "deliveryMode"=>-2, "readAheadAllowed"=>-1, "XMSC_DESTINATION_NAME"=>"REPLY.Q.NAME", "wildcardFormat"=>0, "mdReadEnabled"=>false, "messageBody"=>2, "XMSC_WMQ_BROKER_PUBQ"=>"","XMSC_ADMIN_OBJECT_TYPE"=>33, "priority"=>-2, "encoding"=>273, "brokerVersion"=>0, "version"=>7, "brokerDurSubQueue"=>"SYSTEM.JMS.D.SUBSCRIBER.QUEUE", "XMSC_CONNECTION_TYPE_NAME"=>"com.ibm.msg.client.wmq", "XMSC_WMQ_REPLYTO_STYLE"=>0, "targetClient"=>0, "mdMessageContext"=>0, "XMSC_WMQ_BROKER_PUBQ_QMGR"=>"", "XMSC_CONNECTION_TYPE"=>1}, :jms_timestamp=>1491916141150, :jms_type=>"mcd://xmlnsc/msg_set"}Properties: {"LOG_EventType"=>"REQFINISH", "JMS_IBM_Character_Set"=>"UTF-8", "JMS_IBM_Encoding"=>529, "ESB_Sndr"=>"Adp_name", "JMS_IBM_PutApplType"=>11, "ESB_Rcvr"=>"Srv_name", "JMS_IBM_Format"=>" ", "JMSXDeliveryCount"=>1, "LOG_ReplyQueue"=>"REPLY.Q.NAME", "JMS_IBM_PutTime"=>"13090115", "LOG_EventDate"=>"2017-04-06 15:42:20.532", "LOG_TargetQM"=>"QMNAME", "ESB_TrnId"=>"79acc522-1ac6-11e7-9350-0af37b6b0000", "ESB_MsgFlags"=>"0000000010", "JMS_IBM_MsgType"=>1, "ESB_MsgType"=>"REQUEST", "LOG_ServiceName"=>"Adp_name", "JMSXUserID"=>"ruamdaj ", "ESB_EntryPoint"=>"ADAPTER", "JMSXAppID"=>":\\IBM\\RFHUTIL75\\rfhutilc.exe", "ESB_BrkName"=>"BKNAME", "ESB_ExecGrp"=>"execgrp", "ESB_Operation"=>"GetInfo", "JMS_IBM_PutDate"=>"20170411", "ESB_MsgStatus"=>"SUCCESS", "LOG_TargetQueue"=>"TARGET.Q.NAME", "ESB_ESBId"=>"79acc522-1ac6-11e7-9350-0af37b6b0000"}, :exception=>java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=java.lang.Byte, simple name=Byte, :backtrace=>["org.logstash.Valuefier.convertNonCollection(Valuefier.java:51)", "org.logstash.Valuefier.convert(Valuefier.java:90)", "org.logstash.ConvertedMap.newFromMap(ConvertedMap.java:31)", "org.logstash.Valuefier.convert(Valuefier.java:75)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:128)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)", "org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)", "org.jruby.runtime.Block.yield(Block.java:142)", "org.jruby.RubyHash$13.visit(RubyHash.java:1355)", "org.jruby.RubyHash.visitLimited(RubyHash.java:648)", "org.jruby.RubyHash.visitAll(RubyHash.java:634)", "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)", "org.jruby.RubyHash.each_pairCommon(RubyHash.java:1351)", "org.jruby.RubyHash.each19(RubyHash.java:1342)", "org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)", "org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.IfNode.interpret(IfNode.java:116)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)", "org.jruby.ast.RescueNode.interpret(RescueNode.java:116)", "org.jruby.ast.BeginNode.interpret(BeginNode.java:83)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)", "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)", "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)", "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:300)", "org.jruby.RubyProc.call19(RubyProc.java:281)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:300)", "org.jruby.RubyProc.call19(RubyProc.java:281)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.IfNode.interpret(IfNode.java:116)",
(the entire stack trace does not fit to message)

alexander-milovidov · April 21, 2017, 8:24am

system · May 19, 2017, 8:34am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.