Hi -
I have the message in the below format, I couldn't able to filter it and convert it to JSON or XML.
I'd want to convert this to Key => Value pair of (Name => Value) and store it in ELK.
Could someone please let me know how do I do this. Thanks !
Have you looked into the xml filter?
yep, I tried.
Sorry the message is as below:
The highlighted one has child nodes, I believe that could be the reason, It's not able to parse as a XML message.
I tried converting it to JSON neither doesn't work.
Is there any way around to sort this out ? Please let me know. Thanks !
It’s not able to parse as a XML message.
What's the error message?
If you post the XML as text that we can copy/paste it'll be easier to help.
Thanks for your prompt reply @magnusbaeck
Oh Sorry, Here is the message:
<Response><Terminal><Name>FreeDiskSpace</Name><Value>219201792</Value></Terminal><Terminal><Name>FreeMemContiguous</Name><Value>2208952</Value></Terminal><Terminal><Name>FreeMem</Name><Value>2859015.109375</Value></Terminal><Terminal><Name>CpuLoad</Name><Value><DimSize>4</DimSize><Name>Total</Name><Value>73.3012006900164</Value><Name>Total</Name><Value>66.33462030938836</Value><Name>Total</Name><Value>59.527448701635</Value><Name>Total</Name><Value>55.03308051258238</Value></Value></Terminal></Response>
Here is my logstash configuration:
input {
http_poller {
urls => {
test1 => "http://localhost:8000/tdf/ResourceMonitor"
}
request_timeout => 60
schedule => { cron => "* * * * * UTC"}
codec => "plain"
}
}
filter {
xml {
source => "message"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost:9200"
index => "webservices"
workers => 1
}
}
When I add a filter to convert it to XML, "target = xml", below is the error message I receive:
And it won't even index the documents in Elastic Search
The error message means that the xml.Terminal.Value field has been mapped in a certain way by ES but the document you're currently trying to send to ES requires a different mapping. If you only have test data in your index you can just delete it and try again. Regardless you need to read more about ES mappings and understand why this problem occurred in the first place. Did the schema of the XML document change between the times the http_poller plugin fetched it?
Hi @magnusbaeck -
The schema remains the constant for every run.
The response looks as below:
<Response><Terminal><Name>FreeDiskSpace</Name><Value>219201792</Value></Terminal><Terminal><Name>FreeMemContiguous</Name><Value>2208952</Value></Terminal><Terminal><Name>FreeMem</Name><Value>2859015.109375</Value></Terminal><Terminal><Name>CpuLoad</Name><Value><DimSize>4</DimSize><Name>Total</Name><Value>73.3012006900164</Value><Name>Total</Name><Value>66.33462030938836</Value><Name>Total</Name><Value>59.527448701635</Value><Name>Total</Name><Value>55.03308051258238</Value></Value></Terminal></Response>
Please help me in mapping this fields as key=> value pair. Thanks !
What do you current index mappings look like? Use ES's get mapping API.
Hi @magnusbaeck
Here is the Mappings:
{
"webservices": {
"mappings": {
"logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
Really? Exactly how did you obtain those mappings, and what kind of data did you have in the index?
Hi @magnusbaeck
I've obtained the mapping, with the command "GET webservices/_all/_mapping" (Webservices is the index)
Here is the data I receive it on ES:
Please let me know if you need any additional information. Thanks !
Okay, but that example document contains the non-parsed XML. I'm interested in what the mappings look like when you're getting the mapping conflict.
Yep, when the conflicts are there, it doesn't index the documents on ES, I believe. Thanks !
Surely at least the first document is indexed, otherwise there can't be any conflicts, or?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.