# Not able to parse custom logs having multi line xml

**URL:** <https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743>\
**Category:** Logstash\
**Created:** [November 15, 2017, 12:23pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743 "2017-11-15T12:23:25Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 15, 2017, 12:23pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/1 "2017-11-15T12:23:25Z")

</div>

I have following log file,

```auto
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword 
<properties>
<hostname>crt-mon</hostname>
<date>2016.11.01</date>
<time>01:23:04 CET</time>
<release>11.6</release>
<version>2.1</version>
</properties>

```

and my conf file is,

input {  
file {  
path =\> "D:\mars.log"  
type =\> "test-xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}  
filter {  
grok{  
match =\>{  
"message" =\>"%{DATA:method\_id} %{WORD:method\_type} %{WORD:method} %{GREEDYDATA:data}"  
}  
}  
xml {  
store\_xml =\> "false"  
source =\> "data"  
xpath =\> [  
"/properties/hostname/text()", "hostname",  
"/properties/date/text()", "date",  
"/properties/time/text()", "time",  
"/properties/release/text()", "release",  
"/properties/version/text()", "version"  
]  
}  
mutate {  
rename =\> [  
"[hostname][0]", "hostname",  
"[date][0]", "date",  
"[time][0]", "time",  
"[release][0]", "release",  
"[version][0]", "version"  
]  
}  
}

output {  
elasticsearch {  
index =\> "find"  
hosts =\> ["localhost:9200"]  
}  
stdout { codec =\> rubydebug }  
}

When I'm running the conf file, I'm not getting the xml data, the xml data is only present inside the message tag, but when I keep the whole log file data in single line then I'm able to parse xml data properly. I don't want to keep data in single line, please help me.

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 15, 2017, 12:25pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/2 "2017-11-15T12:25:06Z")

</div>

my log file got messed up above, this is the formatted log file,

5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword  
  
crt-mon  
2016.11.01  
01:23:04 CET  
11.6  
2.1

> Blockquote

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 15, 2017, 2:53pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/3 "2017-11-15T14:53:27Z")

</div>

Is the record shown the full file contents or is the record structure repeated like this?

```auto
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword 
<properties>
<hostname>crt-mon</hostname>
<date>2016.11.01</date>
<time>01:23:04 CET</time>
<release>11.6</release>
<version>2.1</version>
</properties>
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword 
<properties>
<hostname>crt-mon</hostname>
<date>2016.11.01</date>
<time>01:23:04 CET</time>
<release>11.6</release>
<version>2.1</version>
</properties>
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword 
<properties>
<hostname>crt-mon</hostname>
<date>2016.11.01</date>
<time>01:23:04 CET</time>
<release>11.6</release>
<version>2.1</version>
</properties>

```

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 16, 2017, 5:04am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/4 "2017-11-16T05:04:44Z")

</div>

It is not repeated, it contains only until first end tag.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 16, 2017, 8:59am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/5 "2017-11-16T08:59:52Z")

</div>

Use Filebeat multiline. See this thread [Input Json file](https://discuss.elastic.co/t/input-json-file/107781/5)

Its about JSON but the same pitfalls apply to XML.

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 16, 2017, 11:43am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/6 "2017-11-16T11:43:46Z")

</div>

Thanks for your help, but I'm not able to understand that link. can you provide me a example?  
Also, i tried few things and now I'm able to parse the multiline xml data but the problem is i see the data only when i terminate the logstash terminal by pressing cntrl+c (using windows system), otherwise the screen stays on the line as follows,

[2017-11-16T16:40:57,638][INFO][logstash.pipeline] Pipeline main started  
[2017-11-16T16:40:57,851][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 16, 2017, 2:54pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/7 "2017-11-16T14:54:52Z")

</div>

Previous discussions.

> [@Filebeat Process multilne XML](https://discuss.elastic.co/t/filebeat-process-multilne-xml/77761):
>
> I have a log file where I am printing request and response XML bodies. This log file also has some additional lines that are not XML. Example below -- =========================================== \<request\> \<data\> abc \</data\> \</request\> some random lines concerning the processing of request.. fha;js;fkgak;gj;kj;a jgahkg;gaj; Now response follows \<response\> \<output\> def \</output\> \</response\> =================================================== How to use use filebeat multiline fe…

> [@How to configure FileBeat and Logstash to add XML Files in Elasticsearch?](https://discuss.elastic.co/t/how-to-configure-filebeat-and-logstash-to-add-xml-files-in-elasticsearch/86315):
>
> Hi everybody, I'm a beginner here. My own problem is to configure FileBeat and Logstash to add XML Files in Elasticsearch on CentOS 7. I have already install the last version of filebeat, logstash, elasticsearch and Kibana, with the plug-in "elasticsearch-head" in standalone to see inside elasticsearch. And to test my installation, i have successfully add simple log file from CentOS system (/var/log/messages), and see it inside elasticsearch-head plug-in (6 index and 26 shards): And …

As you can see, this kind of thing is really hard to get 100% right. Most of the time the worst problem is that the very last character in the file is not a newline.

Another possibility is to use a python script preprocessor to read a whole file and append a minified XML string to another file that filebeat will tail.  
A partial solution. [python - Remove whitespaces in XML string - Stack Overflow](https://stackoverflow.com/a/3317008/5349531)

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 17, 2017, 5:08am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/8 "2017-11-17T05:08:52Z")

</div>

Thanks again for another reply.  
As i said, I'm able to parse the whole xml data now but there is one small problem.  
When i run the command- logstash -f testxml.CONF, from the command line, i get following lines.

D:\logstash-5.6.3\bin\>logstash -f testxml.CONF  
Picked up \_JAVA\_OPTIONS: -Xmx512M -Xms256M  
Sending Logstash's logs to D:/logstash-5.6.3/logs which is now configured via log4j2.properties  
[2017-11-16T16:40:51,289][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"D:/logstash-5.6.3/modules/fb\_apache/configuration"}  
[2017-11-16T16:40:51,320][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"D:/logstash-5.6.3/modules/netflow/configuration"}  
[2017-11-16T16:40:54,080][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2017-11-16T16:40:54,080][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"}  
[2017-11-16T16:40:54,374][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2017-11-16T16:40:54,593][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2017-11-16T16:40:54,609][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2017-11-16T16:40:54,609][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2017-11-16T16:40:57,090][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
[2017-11-16T16:40:57,638][INFO][logstash.pipeline] Pipeline main started  
[2017-11-16T16:40:57,851][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

But the screen stays there even though i have provided stdout in my conf file, the moment i press cntrl+c, the pipeline stops and only after that i get my output like below,

[2017-11-16T16:40:57,638][INFO][logstash.pipeline] Pipeline main started  
[2017-11-16T16:40:57,851][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2017-11-16T16:43:48,191][WARN][logstash.runner] SIGINT received. Shutting down the agent.  
[2017-11-16T16:43:48,206][WARN][logstash.agent] stopping pipeline {:id=\>"main"}  
{  
"path" =\> "D:\check.xml",  
"hostname" =\> "KHAN",  
"@timestamp" =\> 2017-11-16T11:13:48.831Z,  
"@version" =\> "1",  
"host" =\> "01HW536446",  
"message" =\> "\r\nKHAN\r",  
"type" =\> "test-xml",  
"tags" =\> [  
[0] "multiline"  
]  
}  
Terminate batch job (Y/N)? y

I need to know why the output is not coming before pressing cntrl+c.  
Please help.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 17, 2017, 10:41am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/9 "2017-11-17T10:41:19Z")

</div>

what is the multiline codec configuration?

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 17, 2017, 11:54am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/10 "2017-11-17T11:54:43Z")

</div>

my input looks like this,

input {  
file {  
path =\> "D:\mars.log"  
type =\> "test-xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "nul"  
codec =\> multiline  
{  
pattern =\> "^"  
negate =\> true  
what =\> "previous"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 17, 2017, 2:34pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/11 "2017-11-17T14:34:00Z")

</div>

You may need auto flush.  
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto\_flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto_flush_interval)

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 20, 2017, 5:21am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/12 "2017-11-20T05:21:26Z")

</div>

Thank you so much for your help, I'm able to parse the data completely and properly.

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 24, 2017, 7:09am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/13 "2017-11-24T07:09:14Z")

</div>

Hi, I need one help.  
I want to know what changes i need to make in CONF file if i have repeated xml tags in my xml file as below,

```auto
<properties>
<hostname>crt-moner</hostname>
<date>2016.11.02</date>
<time>01:28:04 CET</time>
<release>11.7</release>
<version>2.2</version>
</properties>
<properties>
<hostname>crt-monerqq</hostname>
<date>2017.11.02</date>
<time>05:28:04 CET</time>
<release>12.7</release>
<version>2.4</version>
</properties>

```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 24, 2017, 9:21am UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/14 "2017-11-24T09:21:25Z")

</div>

Try this:

```auto
input {
  file {
    path => "D:\mars.log"
    type => "test-xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^<properties>"
      negate => true
      what => "previous"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Husain\_Khan](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Husain\_Khan](https://discuss.elastic.co/u/Husain_Khan)\
**Post date:** [November 24, 2017, 12:39pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/15 "2017-11-24T12:39:53Z")

</div>

Thanks alot, it worked 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 22, 2017, 12:40pm UTC](https://discuss.elastic.co/t/not-able-to-parse-custom-logs-having-multi-line-xml/107743/16 "2017-12-22T12:40:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.