# Logstash Message Parsing

**URL:** <https://discuss.elastic.co/t/logstash-message-parsing/98970>\
**Category:** Logstash\
**Created:** [August 31, 2017, 8:32am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970 "2017-08-31T08:32:53Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [August 31, 2017, 8:32am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/1 "2017-08-31T08:32:53Z")

</div>

Hi -  
I have the message in the below format, I couldn't able to filter it and convert it to JSON or XML.  
I'd want to convert this to Key =\> Value pair of (Name =\> Value) and store it in ELK.  
Could someone please let me know how do I do this. Thanks !

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/4/048c52d70b713a8225b635108fc03a4625dd0164.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 31, 2017, 8:56am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/2 "2017-08-31T08:56:17Z")

</div>

Have you looked into the xml filter?

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 1, 2017, 11:29am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/3 "2017-09-01T11:29:08Z")

</div>

yep, I tried.  
Sorry the message is as below:

![image](https://us1.discourse-cdn.com/elastic/original/3X/3/3/3388faff1cc843a5befc1781ff479d66e14862b7.png)

The highlighted one has child nodes, I believe that could be the reason, It's not able to parse as a XML message.  
I tried converting it to JSON neither doesn't work.

Is there any way around to sort this out ? Please let me know. Thanks !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 1, 2017, 11:54am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/4 "2017-09-01T11:54:38Z")

</div>

> It’s not able to parse as a XML message.

What's the error message?

If you post the XML as text that we can copy/paste it'll be easier to help.

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 3, 2017, 8:50am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/5 "2017-09-03T08:50:24Z")

</div>

Thanks for your prompt reply @magnusbaeck  
Oh Sorry, Here is the message:

```
<Response><Terminal><Name>FreeDiskSpace</Name><Value>219201792</Value></Terminal><Terminal><Name>FreeMemContiguous</Name><Value>2208952</Value></Terminal><Terminal><Name>FreeMem</Name><Value>2859015.109375</Value></Terminal><Terminal><Name>CpuLoad</Name><Value><DimSize>4</DimSize><Name>Total</Name><Value>73.3012006900164</Value><Name>Total</Name><Value>66.33462030938836</Value><Name>Total</Name><Value>59.527448701635</Value><Name>Total</Name><Value>55.03308051258238</Value></Value></Terminal></Response>

```

Here is my logstash configuration:

input {  
http\_poller {  
urls =\> {  
test1 =\> "[http://localhost:8000/tdf/ResourceMonitor](http://localhost:8000/tdf/ResourceMonitor)"  
}  
request\_timeout =\> 60  
schedule =\> { cron =\> "\* \* \* \* \* UTC"}  
codec =\> "plain"  
}  
}  
filter {  
xml {  
source =\> "message"  
}  
}   
output {  
stdout {  
codec =\> rubydebug  
}  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "webservices"  
workers =\> 1  
}  
}

When I add a filter to convert it to XML, "target = xml", below is the error message I receive:  
And it won't even index the documents in Elastic Search

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cc613139c492cad59164a81e4318b658bf8b8db.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 3, 2017, 6:35pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/6 "2017-09-03T18:35:26Z")

</div>

The error message means that the xml.Terminal.Value field has been mapped in a certain way by ES but the document you're currently trying to send to ES requires a different mapping. If you only have test data in your index you can just delete it and try again. Regardless you need to read more about ES mappings and understand why this problem occurred in the first place. Did the schema of the XML document change between the times the http\_poller plugin fetched it?

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 4, 2017, 6:48am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/7 "2017-09-04T06:48:24Z")

</div>

Hi @magnusbaeck -  
The schema remains the constant for every run.  
The response looks as below:  
`<Response><Terminal><Name>FreeDiskSpace</Name><Value>219201792</Value></Terminal><Terminal><Name>FreeMemContiguous</Name><Value>2208952</Value></Terminal><Terminal><Name>FreeMem</Name><Value>2859015.109375</Value></Terminal><Terminal><Name>CpuLoad</Name><Value><DimSize>4</DimSize><Name>Total</Name><Value>73.3012006900164</Value><Name>Total</Name><Value>66.33462030938836</Value><Name>Total</Name><Value>59.527448701635</Value><Name>Total</Name><Value>55.03308051258238</Value></Value></Terminal></Response>`

Please help me in mapping this fields as key=\> value pair. Thanks !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 4, 2017, 11:17am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/8 "2017-09-04T11:17:06Z")

</div>

What do you current index mappings look like? Use ES's get mapping API.

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 4, 2017, 11:24am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/9 "2017-09-04T11:24:24Z")

</div>

Hi @magnusbaeck

Here is the Mappings:

{  
"webservices": {  
"mappings": {  
"logs": {  
"properties": {  
"@timestamp": {  
"type": "date"  
},  
"@version": {  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}  
},  
"message": {  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}  
}  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 4, 2017, 5:53pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/10 "2017-09-04T17:53:19Z")

</div>

Really? Exactly how did you obtain those mappings, and what kind of data did you have in the index?

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 5, 2017, 6:54am UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/11 "2017-09-05T06:54:42Z")

</div>

Hi @magnusbaeck

I've obtained the mapping, with the command "GET webservices/\_all/\_mapping" (Webservices is the index)

Here is the data I receive it on ES:

![image](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f23ddc2383a4729d5fedae0b395281deadd71efd.png)

Please let me know if you need any additional information. Thanks !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 5, 2017, 12:04pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/12 "2017-09-05T12:04:33Z")

</div>

Okay, but that example document contains the non-parsed XML. I'm interested in what the mappings look like when you're getting the mapping conflict.

---

<div class="post-metadata">

**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\
**Post date:** [September 5, 2017, 2:46pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/13 "2017-09-05T14:46:47Z")

</div>

Yep, when the conflicts are there, it doesn't index the documents on ES, I believe. Thanks !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 5, 2017, 2:49pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/14 "2017-09-05T14:49:41Z")

</div>

Surely at least the _first_ document is indexed, otherwise there can't be any conflicts, or?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2017, 2:50pm UTC](https://discuss.elastic.co/t/logstash-message-parsing/98970/15 "2017-10-03T14:50:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.