Logstash apply multiple filters based on condition

Elastic Stack Logstash
#1

I have log lines as belows:

11:05:44,924 DEBUG DataFeed:? - Data received: data=TextMessage={
Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]}
JMSReplyTo={null} JMSDeliveryMode={NON_PERSISTENT}
JMSRedelivered={false} JMSCorrelationID={null} JMSType={null}
JMSTimestamp={Tue Aug 30 11:05:44 BST 2016} JMSExpiration={Tue Aug 30
11:06:44 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD}
XT_S_USER={String:someString} APPNAME={String:someFeeName}
XT_BOOK={String:someBook} } Text={..XML Tags..} }

I am trying to apply XML filter on the above log line which fails with:

Error parsing xml with XmlSimple {:source=>"message", :value=>"<XMLMessage was here>", :exception=>#<NoMethodError: undefined methodstart_with?' for nil:NilClass>, :backtrace=>["logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/event.rb:130:in []='", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:infilter'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in multi_filter'", "(eval):41:infilter_func'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in filter_batch'", "org/jruby/RubyArray.java:1613:ineach'", "org/jruby/RubyEnumerable.java:852:in inject'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:infilter_batch'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in worker_loop'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:instart_workers'"], :level=>:warn}`

Logstash Config:

input {
  beats {
    port => 5066
  }
}

filter {
  xml {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "indexNameHere"
  }
}

Also certain lines do not have XML content I want to send them under grok filter how should i achieve this?

TIA!

#2

I suspect that portions of the stacktrace has been removed. Can you please edit your post, paste the stacktrace again, and make sure the stacktrace is formatted as preformatted text? Please also include the configuration.

#3

Added requested changes to original post. Let me know if more info needed.

#4

You need to set the target option if store_xml is true, which is the default. Better validation of these options was introduced in v4.0.0 of the plugin (see PR #26).

#5 
Error parsing xml with XmlSimple {:source=>"message", :value=>"11:06:15,623 DEBUG DataFeed:? - Data received: data=TextMessage={ Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]} JMSReplyTo={null} JMSDeliveryMode={NON_PERSISTENT} JMSRedelivered={false} JMSCorrelationID={null} JMSType={null} JMSTimestamp={Tue Aug 30 11:06:15 BST 2016} JMSExpiration={Tue Aug 30 11:07:15 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD} XT_S_USER={String:someString} APPNAME={String:someAppName} XT_BOOK={String:someBookName} } Text={<?xml version=\"1.0\"?><!DOCTYPE PSRequest><PSRequest><trade><XT_CONVERSIONFLAG>F</XT_CONVERSIONFLAG><Few more XML Tags></trade></PSRequest>} }", :exception=>#<REXML::ParseException: #<REXML::ParseException: Declarations can only occur in the doctype declaration.
    Line: 1
    Position: 3508
    Last 80 unconsumed characters:
    <!DOCTYPE PSRequest><PSRequest><trade><XT_CONVERSIONFLAG>F</XT_CONVERSIONFLAG><XT>
    /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/baseparser.rb:354:in `pull_event'
    /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/baseparser.rb:183:in `pull'
    /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/treeparser.rb:22:in `parse'
    /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:249:in `build'
    /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:43:in `initialize'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:in `filter'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter'
    org/jruby/RubyArray.java:1613:in `each'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter'
    (eval):45:in `filter_func'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch'
    org/jruby/RubyArray.java:1613:in `each'
    org/jruby/RubyEnumerable.java:852:in `inject'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in `filter_batch'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop'
    /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in `start_workers'
    Declarations can only occur in the doctype declaration.
    Line: 1
    Position: 3508
    Last 80 unconsumed characters:
    <!DOCTYPE PSRequest><PSRequest><trade><XT_CONVERSIONFLAG>F</XT_CONVERSIONFLAG><XT>, :backtrace=>["/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/treeparser.rb:95:in `parse'", "/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:249:in `build'", "/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:43:in `initialize'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:in `filter'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):45:in `filter_func'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in `filter_batch'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}
#6

After adding target option as "doc" to logstash config i am getting above error.
EDIT: I think i've figured this out. the xml plugin expects pure XML as log line. But in my case my log line is somewhat like this:

11:05:44,924 DEBUG DataFeed:? - Data received: data=TextMessage={
  Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]}
  JMSReplyTo={null} JMSDeliveryMode={NON_PERSISTENT}
  JMSRedelivered={false} JMSCorrelationID={null} JMSType={null}
  JMSTimestamp={Tue Aug 30 11:05:44 BST 2016} JMSExpiration={Tue Aug 30
  11:06:44 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD}
  XT_S_USER={String:someString} APPNAME={String:someFeeName}
  XT_BOOK={String:someBook} } Text={..XML Tags..} }

Where the XML message appears in between the log line. The moment I sent a pure XML message without any non XML lines it got passed and there were no xmlparsefailures. So know the question is how should we tell xml filter plugin to parse only part of the log line. something like parse everything after Text={'XML message here'}

#7

Use a grok filter to extract the XML to a separate field, then feed that field to the xml filter.

#8

Hi would you be able to share small snippet as to how it could be achieved?
My case is something like this:

if(message.contains(certainText)) {
//apply grok filter and extract xml from it and send it to xml filter and then to ES
} else if(message.contains(someOtherText)) {
//Apply grok filter and extract key value pairs and send it to kv filter and then to ES
} else {
//straight away send that message to ES without any parsing.
}
#9