XML - Illegal Argument Exception

Elastic Stack Logstash
1

Hello Everyone,

I'm unable to dissect my xml file. I'm getting some error of "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [CELL] cannot be changed from type [text] to [ObjectMapper]"}. Please find below my .conf file

input
{
        file
        {
                path => "/home/admin/*_CSSR.xml"
                codec => multiline
                {
                        pattern => "^<moid*>"
                        negate => "true"
                        what => "previous"
                }
        }
}

filter
{
        xml
        {
                source => "message"
                store_xml => true
                target => "parsed"
        }

        dissect                                                                                                                                                                                                            {                                                                                                                                                                                                                          mapping =>
                {
                        "message" => '<moid>%{BSC}/GCELL:LABEL=%{CELL}, CellIndex=%{CellIndex}, CGI=%{CGI}</moid>
<r>%{CELL.ASS.SUCC.RATE}</r>
<r>%{CELL.BSS.CALL.ESTB.SUCC.RATE}</r>
<r>%{CELL.CALL.ESTB.SUCC.RATE}</r>
<r>%{CELL.ESTB.IND.SUCC.RATE.IMM.ASS}</r>
<r>%{CELL.SD.CALL.DROP}</r>
%{?lereste}<mv>'                                                                                                                                                                                                                   }
        }
}

Here is the errors messages:

[WARN ] 2022-09-05 16:39:10.714 [[main]>worker7] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"xml-05-09-2022", :routing=>nil}, {"BSC"=>"BSC04_KANKAN", "CELL"=>"MAN22471D", "CGI"=>"611019C44608A", "CELL.BSS.CALL.ESTB.SUCC.RATE"=>"98.031", "CELL.ASS.SUCC.RATE"=>"100", "tags"=>["multiline", "_xmlparsefailure"], "log"=>{"file"=>{"path"=>"/home/admin/HOST04_pmresult_60_202209051500_202209051600_CSSR.xml"}}, "host"=>{"name"=>"ogn-poc-uat-aiml-cam"}, "message"=>"<moid>BSC04_KANKAN/GCELL:LABEL=MAN22471D, CellIndex=449, CGI=611019C44608A</moid>\n<r>100</r>\n<r>98.031</r>\n<r>92.857</r>\n<r>98.031</r>\n<r>0</r>\n<sf>false</sf>\n</mv>\n<mv>", "@timestamp"=>2022-09-05T16:39:09.689004Z, "event"=>{"original"=>"<moid>BSC04_KANKAN/GCELL:LABEL=MAN22471D, CellIndex=449, CGI=611019C44608A</moid>\n<r>100</r>\n<r>98.031</r>\n<r>92.857</r>\n<r>98.031</r>\n<r>0</r>\n<sf>false</sf>\n</mv>\n<mv>"}, "@version"=>"1", "CELL.CALL.ESTB.SUCC.RATE"=>"92.857", "CELL.ESTB.IND.SUCC.RATE.IMM.ASS"=>"98.031", "CELL.SD.CALL.DROP"=>"0", "CellIndex"=>"449"}], :response=>{"index"=>{"_index"=>"xml-05-09-2022", "_id"=>"gM-FDoMB5KKfgEMDJsEl", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [CELL] cannot be changed from type [text] to [ObjectMapper]"}}}}
[WARN ] 2022-09-05 16:39:11.853 [[main]>worker5] xml - Error parsing xml with XmlSimple {:source=>"message", :value=>"<moid>BSC04_KANKAN/GCELL:LABEL=KER22464A, CellIndex=262, CGI=611019C4A6041</moid>\n<r>100</r>\n<r>98.305</r>\n<r>91.304</r>\n<r>98.305</r>\n<r>0</r>\n<sf>false</sf>\n</mv>\n</mi>\n</md>\n<mff>\n<ts>202209051600</ts>\n</mff>\n</mdc>", :exception=>#<REXML::ParseException: #<RuntimeError: attempted adding second root element to document>
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:197:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:1931:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/child.rb:22:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/parent.rb:14:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:328:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:1929:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:733:in `add_element'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:210:in `add_element'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/parsers/treeparser.rb:34:in `parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:448:in `build'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:101:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:979:in `parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:164:in `xml_in'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:203:in `xml_in'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.1.3/lib/logstash/filters/xml.rb:195:in `filter'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'
org/jruby/RubyArray.java:1821:in `each'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'
org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in `block in start_workers'
...
attempted adding second root element to document
Line: 2
Position: 85
Last 80 unconsumed characters:
>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/parsers/treeparser.rb:96:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:448:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:101:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:979:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:164:in `xml_in'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:203:in `xml_in'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.1.3/lib/logstash/filters/xml.rb:195:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in `block in start_workers'"]}

Because the pattern is repeated (as it is xml file), I see the ParseException, saying : #<RuntimeError: attempted adding second root element to document> .
I don't know how to fix this issue.

Your help is really appreciate.
Thanks in advance.

2

That is saying that there are more than one root elements in the document you are trying to parse. The xml filter cannot support that. Looking at your message....

You have the root element <moid>, followed by multiple additional root elements <r>, followed by root element <sf> etc. You could wrap everything in an additional element.

mutate {
    gsub => [
        "message", "^", "<foo>", 
        "message", "$", "</foo>"
    ]
}

Then you will have a single root element <foo>.

3

Thanks a lot for your reply.

I tried your solution but I have almost same issue.

Here the errors messages:

[WARN ] 2022-09-05 19:10:28.711 [[main]>worker7] Dissector - Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<moid>%{BSC}/GCELL:LABEL=%{CELL}, CellIndex=%{CellIndex}, CGI=%{CGI}</moid>\n<r>%{CELL.ASS.SUCC.RATE}</r>\n<r>%{CELL.BSS.CALL.ESTB.SUCC.RATE}</r>\n<r>%{CELL.CALL.ESTB.SUCC.RATE}</r>\n<r>%{CELL.ESTB.IND.SUCC.RATE.IMM.ASS}</r>\n<r>%{CELL.SD.CALL.DROP}</r>\n%{?lereste}<mv>", "event"=>{"@timestamp"=>2022-09-05T19:10:25.282820Z, "tags"=>["multiline", "_xmlparsefailure", "_dissectfailure"], "message"=>"<global><moid>BSC04_KANKAN/GCELL:LABEL=MAN20675B, CellIndex=309, CGI=611019C481A60</moid></global>\n<global><r>99.711</r></global>\n<global><r>94.104</r></global>\n<global><r>82.933</r></global>\n<global><r>94.439</r></global>\n<global><r>0.066</r></global>\n<global><sf>false</sf></global>\n<global></mv></global>\n<global><mv></global>", "event"=>{"original"=>"<moid>BSC04_KANKAN/GCELL:LABEL=MAN20675B, CellIndex=309, CGI=611019C481A60</moid>\n<r>99.711</r>\n<r>94.104</r>\n<r>82.933</r>\n<r>94.439</r>\n<r>0.066</r>\n<sf>false</sf>\n</mv>\n<mv>"}, "log"=>{"file"=>{"path"=>"/home/admin/HOST04_pmresult_60_202209051700_202209051800_CSSR.xml"}}, "host"=>{"name"=>"ogn-poc-uat-aiml-cam"}, "@version"=>"1"}}
[WARN ] 2022-09-05 19:10:28.711 [[main]>worker5] xml - Error parsing xml with XmlSimple {:source=>"message", :value=>"<moid>BSC04_KANKAN/GCELL:LABEL=MAN22477C, CellIndex=404, CGI=611019C4160C5</moid>\n<r>100</r>\n<r>100</r>\n<r>96.053</r>\n<r>100</r>\n<r>0</r>\n<sf>false</sf>\n</mv>\n<mv>", :exception=>#<REXML::ParseException: #<RuntimeError: attempted adding second root element to document>
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:197:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:1931:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/child.rb:22:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/parent.rb:14:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:328:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:1929:in `add'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/element.rb:733:in `add_element'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:210:in `add_element'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/parsers/treeparser.rb:34:in `parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:448:in `build'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rexml-3.2.5/lib/rexml/document.rb:101:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:979:in `parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:164:in `xml_in'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.9/lib/xmlsimple.rb:203:in `xml_in'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.1.3/lib/logstash/filters/xml.rb:195:in `filter'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'
org/jruby/RubyArray.java:1821:in `each'
/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'
org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in `block in start_workers'
...
attempted adding second root element to document
Line: 2
Position: 85

Any idea?

4

The value of the [message] field has not changed, so you will get the same errors. Did you add the mutate filter before the xml filter?

5

The value of the [message] field has not changed, so you will get the same errors. Did you add the mutate filter before the xml filter?

Yes, I did this manner, but I think it is also inserting my global wrapper as you can see on the first line of the output shared up. Why the message=> on the first two line are differente from the value.

I notice on the 1st line that the "message" => is different from the value you pointed out while trying to parse. Why this kind of behoviour or i'm misunderstanding.

6

Impossible to say without seeing your configuration.

7

Hello @Badger ,
I made many work arround but nothing work. I made a little change in the conf file and getting some error as shown in this txt file.

Here is my actual .conf file

Thanks in advance.

8

Hello @Badger ,

Until now I don't know how to get out from this xml issue. have you get some time for what I shared.

Best Regards.

9

There are two different errors in that log.

xml - Error parsing xml with XmlSimple {:source=>"message", :value=>"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n\n202209071000\n\n\n\n202209071000\n60\nCELL.ASS.SUCC.RATE\nCELL.BSS.CALL.ESTB.SUCC.RATE\nCELL.CALL.ESTB.SUCC.RATE\nCELL.ESTB.IND.SUCC.RATE.IMM.ASS\nCELL.SD.CALL.DROP", :exception=>#<REXML::ParseException: No close tag for /mdc/md/mi

As it says, the <mi> element is never closed.

The second error is

Dissector - Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"\n%{BSC}/GCELL:LABEL=%{CELL}, CellIndex=%{CellIndex}, CGI=%{CGI}\n%{CELL.ASS.SUCC.RATE}\n%{CELL.BSS.CALL.ESTB.SUCC.RATE}\n%{CELL.CALL.ESTB.SUCC.RATE}\n%{CELL.ESTB.IND.SUCC.RATE.IMM.ASS}\n%{CELL.SD.CALL.DROP}\n%{}\n", "event"=>{"event"=>{"original"=>"\nBSC04_KANKAN/GCELL:LABEL=KRS20705A, CellIndex=526, CGI=611019C4C1B8B\n100\n96.848\n98.802\n96.848\n0\nfalse\n"}, "message"=>{"sf"=>["false"], "moid"=>["BSC04_KANKAN/GCELL:LABEL=KRS20705A, CellIndex=526, CGI=611019C4C1B8B"], "r"=>["100", "96.848", "98.802", "96.848", "0"]}

In this case the xml has successfully parsed and overwritten the [message] field, so the dissect pattern does not match it.

10

OK. Noted!!! And thank a lot for your time.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.