I am working with local filebeat and logstash and elasticsearch and kibana on a server. I need to convert the information that comes to me from the logs from xml to json. My configuration works when it reads a file with log in xml but when I use a log generator and send them this conversion is never done and it shows me the following message:
MI LOG
<log4j:event logger="LogGeneratorApp.MainForm" timestamp="1662748734048" level="INFO" thread="1">log4j:messageThis is a test log message</log4j:message>log4j:properties<log4j:data name="log4net:UserName" value="TE\60085367" /><log4j:data name="log4jmachinename" value="TAMP00043541" /><log4j:data name="log4japp" value="LogGeneratorApp.exe" /><log4j:data name="log4net:HostName" value="TAMP00043541" /></log4j:properties><log4j:locationInfo class="LogGeneratorApp.MainForm" method="SendToLog" file="" line="0" /></log4j:event>
Y EL ERROR EN CONSOLA ES:
[2022-09-09T13:39:28,195][WARN ][logstash.filters.xml ][main][119f194f3af434e5618422bbe0983095b06acb780727af330f680c5335d16fe4] Error parsing xml with XmlSimple {:source=>"message", :value=>"<log4j:event logger="LogGeneratorApp.MainForm" timestamp="1662748761483" level="INFO" thread="1">log4j:messageTIMER Generated : This is a test log message</log4j:message>log4j:properties<log4j:data name="log4net:UserName" value="TE\60085367" /><log4j:data name="log4jmachinename" value="TAMP00043541" /><log4j:data name="log4japp" value="LogGeneratorApp.exe" /><log4j:data name="log4net:HostName" value="TAMP00043541" /></log4j:properties><log4j:locationInfo class="LogGeneratorApp.MainForm" method="SendToLog" file="" line="0" /></log4j:event>", :exception=>#<REXML::UndefinedNamespaceException: Undefined prefix log4j found>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/baseparser.rb:414:in `block in pull_event'", "org/jruby/ext/set/RubySet.java:628:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/baseparser.rb:412:in `pull_event'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/baseparser.rb:185:in `pull'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/treeparser.rb:23:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:288:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:45:in `initialize'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.8/lib/xmlsimple.rb:979:in `parse'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.8/lib/xmlsimple.rb:164:in `xml_in'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.8/lib/xmlsimple.rb:203:in `xml_in'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.1.2/lib/logstash/filters/xml.rb:195:in `filter'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'", "org/jruby/RubyArray.java:1820:in `each'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'", "C:/Users/60085367/Documents/Parsing090922/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:295:in `block in start_workers'"]}
MY LOGSTASH
input {
beta {port => 5044}
}
filter{
xml{
source => "message"
target => "doc"
}
}
output{
stdout {codec => rubydebug]
}
MY FILEBEAT
filebeat.inputs:
- type: filestream
id: my-filestream
enable: true
path: C:\..........\logs\*
multiline.pattern: ^<log4j:event.*
multiline.negate: true
multiline.match: after
File Modules default settings
Elasticsearch template default setting
Output.logstash:
host: ["0.0.0.0:5044"]
Processors default