# Multiline XML processing using logstash

**URL:** https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915
**Category:** Logstash
**Created:** [January 3, 2018, 12:08pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915 "2018-01-03T12:08:24Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)
#### Post date: [January 3, 2018, 12:08pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/1 "2018-01-03T12:08:24Z")

</div>

I have multi line XML to process more than 1000 lines. I am generating single message out of it.  
When using Logstash 2.4 multiline filter plugin was working perfectly. But as its been deprecated in 6.1.

Therefore modified config to use this as codec multiline for 6.1 ,  
input {  
stdin {  
type =\> "xml file path"  
codec =\> multiline {  
pattern =\>`"^<response>"`  
negate =\> true  
what =\> "previous"  
}  
}

this just process half of the files.

later added max\_lines & max\_bytes seeing reference document

stdin {  
type =\> " xml path"  
codec =\> multiline {  
pattern =\>`"^<response>"`  
negate =\> true  
what =\> "previous"  
max\_bytes =\> "180 mb"  
max\_lines =\> 1000  
}  
Above config does nothing and simply terminates the pipline saying...  
[INFO][logstash.pipeline] Pipeline terminated

Any help?

---

<div class="post-metadata">

### Author: ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)
#### Post date: [January 4, 2018, 8:19am UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/2 "2018-01-04T08:19:47Z")

</div>

Any help in this regard???

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 5, 2018, 9:55am UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/3 "2018-01-05T09:55:11Z")

</div>

What comes prior to "Pipeline terminated"?

Seems easier to just use an exec input to cat the input file.

---

<div class="post-metadata">

### Author: ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)
#### Post date: [January 5, 2018, 11:31am UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/4 "2018-01-05T11:31:47Z")

</div>

Log to refer:  
[2018-01-05T14:09:26,315][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/netflow/configuration"}  
[2018-01-05T14:09:26,498][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2018-01-05T14:09:27,018][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.1.1"}  
[2018-01-05T14:09:27,365][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2018-01-05T14:09:34,368][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>500, "pipeline.batch.delay"=\>10, "pipeline.max\_inflight"=\>1000, :thread=\>"#\<Thread:0x5b60c5d7 run\>"}  
[2018-01-05T14:09:34,433][INFO][logstash.pipeline] Pipeline started {"[pipeline.id](http://pipeline.id)"=\>"main"}  
[2018-01-05T14:09:34,541][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}  
[2018-01-05T14:09:35,624][INFO][logstash.pipeline] Pipeline terminated {"[pipeline.id](http://pipeline.id)"=\>"main"}  
[2018-01-05T14:37:40,821][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/fb\_apache/configuration"}  
[2018-01-05T14:37:40,836][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/netflow/configuration"}  
[2018-01-05T14:37:41,061][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2018-01-05T14:37:41,545][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.1.1"}  
[2018-01-05T14:37:42,224][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2018-01-05T14:37:49,735][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>500, "pipeline.batch.delay"=\>10, "pipeline.max\_inflight"=\>1000, :thread=\>"#\<Thread:0x22025f5 run\>"}  
[2018-01-05T14:37:49,827][INFO][logstash.pipeline] Pipeline started {"[pipeline.id](http://pipeline.id)"=\>"main"}  
[2018-01-05T14:37:49,953][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}  
[2018-01-05T14:37:51,022][INFO][logstash.pipeline] Pipeline terminated {"[pipeline.id](http://pipeline.id)"=\>"main"}

How to use exec input to cat the input file. I am using xpath to select some values from the tags of XML file.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 5, 2018, 12:07pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/5 "2018-01-05T12:07:38Z")

</div>

> How to use exec input to cat the input file.

You use the exec input to run "cat filename.xml".

---

<div class="post-metadata">

### Author: ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)
#### Post date: [January 5, 2018, 12:34pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/6 "2018-01-05T12:34:25Z")

</div>

Sorry for coming again. I am not understanding how to use cat here on Windows.

I am running logstash to process xml file once : logstash -f conf \< xml file

Therefore stdin input i used. so now as multiline codec is deprecated from filter so added it to input.

Please clarify.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 5, 2018, 9:18pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/7 "2018-01-05T21:18:27Z")

</div>

Oh, this is Windows. Well, use whatever equivalent command that's available to you.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 2, 2018, 9:19pm UTC](https://discuss.elastic.co/t/multiline-xml-processing-using-logstash/113915/8 "2018-02-02T21:19:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.