# Logstash apply multiple filters based on condition **URL:** \ **Category:** Logstash\ **Created:** [August 30, 2016, 11:34am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299 "2016-08-30T11:34:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Post date:** [August 30, 2016, 11:34am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/1 "2016-08-30T11:34:00Z")
I have log lines as belows: > 11:05:44,924 DEBUG DataFeed:? - Data received: data=TextMessage={ > Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]} > JMSReplyTo={null} JMSDeliveryMode={NON\_PERSISTENT} > JMSRedelivered={false} JMSCorrelationID={null} JMSType={null} > JMSTimestamp={Tue Aug 30 11:05:44 BST 2016} JMSExpiration={Tue Aug 30 > 11:06:44 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD} > XT\_S\_USER={String:someString} APPNAME={String:someFeeName} > XT\_BOOK={String:someBook} } Text={..XML Tags..} } I am trying to apply XML filter on the above log line which fails with: `Error parsing xml with XmlSimple {:source=>"message", :value=>"", :exception=>#, :backtrace=\>["logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/event.rb:130:in `[]='", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:in `filter'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):41:in `filter\_func'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in `filter\_batch'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in `start\_workers'"], :level=\>:warn}` Logstash Config: ``` input { beats { port => 5066 } } filter { xml { source => "message" } } output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "indexNameHere" } } ``` Also certain lines do not have XML content I want to send them under grok filter how should i achieve this? TIA! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2016, 11:52am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/2 "2016-08-30T11:52:20Z")
I suspect that portions of the stacktrace has been removed. Can you please edit your post, paste the stacktrace again, and make sure the stacktrace is formatted as preformatted text? Please also include the configuration. ---
**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Post date:** [August 30, 2016, 2:34pm UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/3 "2016-08-30T14:34:07Z")
Added requested changes to original post. Let me know if more info needed. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2016, 5:45pm UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/4 "2016-08-30T17:45:23Z")
You need to set the `target` option if `store_xml` is true, which is the default. Better validation of these options was introduced in v4.0.0 of the plugin (see [PR #26](https://github.com/logstash-plugins/logstash-filter-xml/pull/26)). ---
**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Post date:** [August 31, 2016, 7:35am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/5 "2016-08-31T07:35:49Z")
``` Error parsing xml with XmlSimple {:source=>"message", :value=>"11:06:15,623 DEBUG DataFeed:? - Data received: data=TextMessage={ Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]} JMSReplyTo={null} JMSDeliveryMode={NON_PERSISTENT} JMSRedelivered={false} JMSCorrelationID={null} JMSType={null} JMSTimestamp={Tue Aug 30 11:06:15 BST 2016} JMSExpiration={Tue Aug 30 11:07:15 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD} XT_S_USER={String:someString} APPNAME={String:someAppName} XT_BOOK={String:someBookName} } Text={F} }", :exception=>#F /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/baseparser.rb:354:in `pull_event' /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/baseparser.rb:183:in `pull' /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/treeparser.rb:22:in `parse' /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:249:in `build' /logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:43:in `initialize' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:in `filter' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter' org/jruby/RubyArray.java:1613:in `each' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter' (eval):45:in `filter_func' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch' org/jruby/RubyArray.java:1613:in `each' org/jruby/RubyEnumerable.java:852:in `inject' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in `filter_batch' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop' /logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in `start_workers' Declarations can only occur in the doctype declaration. Line: 1 Position: 3508 Last 80 unconsumed characters: F, :backtrace=>["/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/parsers/treeparser.rb:95:in `parse'", "/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:249:in `build'", "/logstash/2.3.2-build001/common/vendor/jruby/lib/ruby/1.9/rexml/document.rb:43:in `initialize'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-2.1.4/lib/logstash/filters/xml.rb:166:in `filter'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):45:in `filter_func'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in `filter_batch'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "/logstash/2.3.2/common/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn} ``` ---
**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Post date:** [August 31, 2016, 7:36am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/6 "2016-08-31T07:36:26Z")
After adding target option as "doc" to logstash config i am getting above error. EDIT: I think i've figured this out. the xml plugin expects pure XML as log line. But in my case my log line is somewhat like this: ``` 11:05:44,924 DEBUG DataFeed:? - Data received: data=TextMessage={ Header={ JMSMessageID={ID:someId} JMSDestination={Topic[someTopic]} JMSReplyTo={null} JMSDeliveryMode={NON_PERSISTENT} JMSRedelivered={false} JMSCorrelationID={null} JMSType={null} JMSTimestamp={Tue Aug 30 11:05:44 BST 2016} JMSExpiration={Tue Aug 30 11:06:44 BST 2016} JMSPriority={4} } Properties={ ACTION={String:ADD} XT_S_USER={String:someString} APPNAME={String:someFeeName} XT_BOOK={String:someBook} } Text={..XML Tags..} } ``` Where the XML message appears in between the log line. The moment I sent a pure XML message without any non XML lines it got passed and there were no xmlparsefailures. So know the question is how should we tell xml filter plugin to parse only part of the log line. something like parse everything after **Text={'XML message here'}** ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 31, 2016, 12:27pm UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/7 "2016-08-31T12:27:28Z")
Use a grok filter to extract the XML to a separate field, then feed that field to the xml filter. ---
**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Post date:** [August 31, 2016, 3:10pm UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/8 "2016-08-31T15:10:51Z")
Hi would you be able to share small snippet as to how it could be achieved? My case is something like this: ``` if(message.contains(certainText)) { //apply grok filter and extract xml from it and send it to xml filter and then to ES } else if(message.contains(someOtherText)) { //Apply grok filter and extract key value pairs and send it to kv filter and then to ES } else { //straight away send that message to ES without any parsing. } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:40am UTC](https://discuss.elastic.co/t/logstash-apply-multiple-filters-based-on-condition/59299/9 "2017-07-06T04:40:35Z")