# Pasrse data between repeated xml tags

**URL:** https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270
**Category:** Logstash
**Created:** [March 14, 2019, 7:49am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270 "2019-03-14T07:49:37Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 14, 2019, 7:49am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/1 "2019-03-14T07:49:37Z")

</div>

I have a xml file that has repeated tags and i need to extract the fields between those tags , in below mentioned xml data I want to get the data between the job\_list tags:

\<?xml version='1.0'?\>

\<job\_info xmlns:xsd="[https://schemas/qstat.xsd](https://schemas/qstat.xsd)"\>  
\<queue\_info\>  
\<job\_list state="running"\>  
\<JB\_job\_number\>148635539\</JB\_job\_number\>  
\<JAT\_prio\>0.51090\</JAT\_prio\>  
\<JAT\_ntix\>1.00000\</JAT\_ntix\>  
\<JB\_nppri\>0.45090\</JB\_nppri\>  
\<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>400G\</hard\_request\>  
\<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>  
\<job\_binding\>NONE\</job\_binding\>  
\</job\_list\>  
\<job\_list state="running"\>  
\<JB\_job\_number\>148711350\</JB\_job\_number\>  
\<JAT\_prio\>0.51090\</JAT\_prio\>  
\<JAT\_ntix\>1.00000\</JAT\_ntix\>  
\<JB\_nppri\>0.45090\</JB\_nppri\>  
\<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>900G\</hard\_request\>  
\<hard\_request name="h\_rt" resource\_contribution="0.000000"\>518400\</hard\_request\>  
\<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>  
\<job\_binding\>NONE\</job\_binding\>  
\</job\_list\>  
\</queue\_info\>  
\<job\_info\>  
\<job\_list state="pending"\>  
\<JB\_job\_number\>133460680\</JB\_job\_number\>  
\<JAT\_prio\>0.00000\</JAT\_prio\>  
\<JAT\_ntix\>0.00000\</JAT\_ntix\>  
\<JB\_nppri\>0.00000\</JB\_nppri\>  
\<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>1000M\</hard\_request\>  
\<hard\_req\_queue\>all.q\</hard\_req\_queue\>  
\<job\_binding\>NONE\</job\_binding\>  
\</job\_list\>  
\<job\_list state="pending"\>  
\<JB\_job\_number\>140091542\</JB\_job\_number\>  
\<JAT\_prio\>0.00000\</JAT\_prio\>  
\<JAT\_ntix\>0.00000\</JAT\_ntix\>  
\<JB\_nppri\>0.00000\</JB\_nppri\>  
\<hard\_request name="arch" resource\_contribution="0.000000"\>lx\*\</hard\_request\>  
\<hard\_req\_queue\>all.q\</hard\_req\_queue\>  
\<job\_binding\>NONE\</job\_binding\>  
\</job\_list\>  
\</job\_info\>  
\</job\_info\>

I tried with below conf but did not seem to work :  
input  
{  
file  
{  
path =\> "/usr/share/logstash/qstat.cache"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
type =\> "xml"  
codec =\> multiline {  
pattern =\> "^\</job\_list\>"  
negate =\> "true"  
what =\> "previous"  
}  
}  
}  
filter  
{  
xml  
{  
source =\> "message"  
store\_xml =\> false  
#target =\> "job\_list"  
xpath =\> [  
"/job\_list/@state", "Job\_State",  
"/job\_list/JB\_job\_number/text()", "Job\_Number",  
"/job\_list/JAT\_prio/text()", "priority",  
"/job\_list/JAT\_ntix/text()", "ntix",  
"/job\_list/JB\_nppri/text()", "nppri",  
"/job\_list/hard\_req\_queue/text()", "hard\_req\_queue",  
"/job\_list/binding/text()", "binding"  
]  
}

}

output  
{  
stdout  
{  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 14, 2019, 2:02pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/2 "2019-03-14T14:02:04Z")

</div>

> [@sverma](#):
>
> codec =\> multiline {  
> pattern =\> "^\</job\_list\>"  
> negate =\> "true"  
> what =\> "previous"  
> }

This will not result in valid XML. I suggest you consume the entire file as a single event using a multiline codec with a pattern that never matches. For example

```
codec => multiline { pattern => "^Spalanzani" what => "previous" negate => true auto_flush_interval => 2 }

```

You will then be able to parse elements from /job\_info/job\_list or /queue\_info/job\_list.

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 14, 2019, 3:14pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/3 "2019-03-14T15:14:40Z")

</div>

Thanks for the suggestion will try out and come back , however I did try something similar but was getting "multiline\_codec\_max\_lines\_reached" error, my original xml has 20631298 lines and it can be even more at some time.  
Is there a way to unlimit the number of lines as I cannot predict how many lines would be there

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 14, 2019, 3:26pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/4 "2019-03-14T15:26:13Z")

</div>

what is the purpose of "auto\_flush\_interval" here in the codec, can you please explain on this.

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 14, 2019, 3:51pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/5 "2019-03-14T15:51:02Z")

</div>

After doing suggested change in codec and xpath as below :  
xpath =\> [  
"/job\_info/queue\_info/job\_list/@state", "Job\_State",  
"/job\_info/queue\_info/job\_list/JB\_job\_number/text()", "Job\_Number",  
"/job\_info/queue\_info/job\_list/JAT\_prio/text()", "priority",  
"/job\_info/queue\_info/job\_list/JAT\_ntix/text()", "ntix",  
"/job\_info/queue\_info/job\_list/JB\_nppri/text()", "nppri",  
"/job\_info/queue\_info/job\_list/hard\_req\_queue/text()", "hard\_req\_queue",  
"/job\_info/queue\_info/job\_list/job\_binding/text()", "binding",  
"/job\_info/job\_info/job\_list/@state", "Job\_State",  
"/job\_info/job\_info/job\_list/JB\_job\_number/text()", "Job\_Number",  
"/job\_info/job\_info/job\_list/JAT\_prio/text()", "priority",  
"/job\_info/job\_info/job\_list/JAT\_ntix/text()", "ntix",  
"/job\_info/job\_info/job\_list/JB\_nppri/text()", "nppri",  
"/job\_info/job\_info/job\_list/hard\_req\_queue/text()", "hard\_req\_queue",  
"/job\_info/job\_info/job\_list/job\_binding/text()", "binding"  
]

When I keep auto\_flush\_interval =\> 2 then I get data for my \<job\_list state = running \> tags as below  
{  
"host" =\> "myelkhost",  
"@version" =\> "1",  
"Job\_State" =\> ,  
"ntix" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"path" =\> "/usr/share/logstash/qstat.cache",  
"priority" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"hard\_req\_queue" =\> [  
[0] "all.q",  
[1] "all.q"  
],  
"Job\_Number" =\> [  
[0] "133460680",  
[1] "140091542"  
],  
"binding" =\> ,  
"type" =\> "xml",  
"nppri" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"@timestamp" =\> 2019-03-14T15:34:17.860Z,  
"tags" =\> [  
[0] "multiline"  
],  
"message" =\> "\<?xml version='1.0'?\>\n\<job\_info xmlns:xsd="[https://schemas/qstat.xsd](https://schemas/qstat.xsd)"\>\n \<queue\_info\>\n \<job\_list state="running"\>\n \<JB\_job\_number\>148635539\</JB\_job\_number\>\n \<JAT\_prio\>0.51090\</JAT\_prio\>\n \<JAT\_ntix\>1.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.45090\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>400G\</hard\_request\>\n \<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \<job\_list state="running"\>\n \<JB\_job\_number\>148711350\</JB\_job\_number\>\n \<JAT\_prio\>0.51090\</JAT\_prio\>\n \<JAT\_ntix\>1.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.45090\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>900G\</hard\_request\>\n \<hard\_request name="h\_rt" resource\_contribution="0.000000"\>518400\</hard\_request\>\n \<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</queue\_info\>\n \<job\_info\>\n \<job\_list state="pending"\>\n \<JB\_job\_number\>133460680\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>1000M\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \<job\_list state="pending"\>\n \<JB\_job\_number\>140091542\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="arch" resource\_contribution="0.000000"\>lx\*\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</job\_info\>\n\</job\_info\>"

and when i keep auto\_flush\_interval =\> i get data for my \<job\_list state = pending \> tags as below  
{  
"nppri" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"tags" =\> [  
[0] "multiline"  
],  
"type" =\> "xml",  
"message" =\> "\<?xml version='1.0'?\>\n\<job\_info xmlns:xsd="[https://schemas/qstat.xsd](https://schemas/qstat.xsd)"\>\n \<queue\_info\>\n \<job\_list state="running"\>\n \<JB\_job\_number\>148635539\</JB\_job\_number\>\n \<JAT\_prio\>0.51090\</JAT\_prio\>\n \<JAT\_ntix\>1.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.45090\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>400G\</hard\_request\>\n \<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \<job\_list state="running"\>\n \<JB\_job\_number\>148711350\</JB\_job\_number\>\n \<JAT\_prio\>0.51090\</JAT\_prio\>\n \<JAT\_ntix\>1.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.45090\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>900G\</hard\_request\>\n \<hard\_request name="h\_rt" resource\_contribution="0.000000"\>518400\</hard\_request\>\n \<hard\_req\_queue\>largemem.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</queue\_info\>\n \<job\_info\>\n \<job\_list state="pending"\>\n \<JB\_job\_number\>133460680\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="mem\_avail" resource\_contribution="0.000000"\>1000M\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \<job\_list state="pending"\>\n \<JB\_job\_number\>140091542\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="arch" resource\_contribution="0.000000"\>lx\*\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</job\_info\>\n\</job\_info\>",  
"@timestamp" =\> 2019-03-14T15:43:54.868Z,  
"host" =\> "myelkhost",  
"@version" =\> "1",  
"Job\_State" =\> [  
[0] "pending",  
[1] "pending"  
],  
"binding" =\> [  
[0] "NONE",  
[1] "NONE"  
],  
"path" =\> "/usr/share/logstash/qstat.cache",  
"hard\_req\_queue" =\> [  
[0] "all.q",  
[1] "all.q"  
],  
"priority" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"ntix" =\> [  
[0] "0.00000",  
[1] "0.00000"  
],  
"Job\_Number" =\> [  
[0] "133460680",  
[1] "140091542"  
]  
}

But I do not get data for all the 4 running and pending status tags at same time.

I am running below command to test my pipeline

# /usr/share/logstash/bin/logstash -f /usr/share/logstash/xml-pipeline.conf\_ask --config.reload.automatic --path.settings /etc/logstash --path.data /usr/share/logstash/qcache/data

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 14, 2019, 4:10pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/6 "2019-03-14T16:10:32Z")

</div>

auto\_flush\_interval tells the codec that if it does not get a match to the regexp for 2 seconds then to flush whatever it has read onto the pipeline as an event. When using a pattern that never matches this is the only way to create an event.

If you have a 20 million line file than I suggest ingesting each job\_info element separately, as you originally tried to do. Use a codec with what =\> "next"

```
codec => multiline { pattern => "^</job_list>" what => "next" negate => true }

```

Then modify the message to remove anything you do not need. I would have expected to be able to do this with mutate+gsub, but I cannot get it to work, so I used ruby

```
    ruby {
        code => '
            m = event.get("message").scan(/<job_list.*/m)
            if m == []
                event.cancel
            else
                event.set("message", m)
            end
        '
    }
    xml { source => "message" target=> "theXML" store_xml => true }
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 16, 2019, 2:40pm UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/7 "2019-03-16T14:40:34Z")

</div>

It can be done with mutate+gsub using

```
 mutate { gsub => ["message", "(?m).*(<job_list)", "\1"] }
```

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 17, 2019, 2:35am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/8 "2019-03-17T02:35:51Z")

</div>

Thanks for the answer, mutate filter seems working fine and is extracting the tags between the job\_list, but when I test the pipline I only get result for last job\_list tag rest all I do not see, is it overriding, with the sample xml file that I pasted earlier I ran below logstash config :

input  
{  
file  
{  
path =\> "/usr/share/logstash/qstat.cache"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
type =\> "xml"  
codec =\> multiline { pattern =\> "^\</job\_list\>" what =\> "next" negate =\> true }  
}  
}  
filter  
{  
mutate { gsub =\> ["message", "(?m).\*(\<job\_list)", "\1"] }

xml  
{  
source =\> "message"  
store\_xml =\> false  
xpath =\> [  
"/job\_list/@state", "Job\_State",  
"/job\_list/JB\_job\_number/text()", "Job\_Number",  
"/job\_list/JAT\_prio/text()", "priority",  
"/job\_list/JAT\_ntix/text()", "ntix",  
"/job\_list/JB\_nppri/text()", "nppri",  
"/job\_list/hard\_req\_queue/text()", "hard\_req\_queue",  
"/job\_list/job\_binding/text()", "binding"  
]  
}  
}  
output  
{ stdout { codec =\> rubydebug }}

I expected all the four job\_list tags info to be printed at my stdout but I got the output for only the last tag after doing "control+C" , below is the pipeline run :

# /usr/share/logstash/bin/logstash -f /usr/share/logstash/qstat-pipeline.conf --config.reload.automatic --path.settings /etc/logstash --path.data /usr/share/logstash/qcache/data

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties  
[2019-03-17T08:04:05,585][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-03-17T08:04:05,606][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.6.1"}  
[2019-03-17T08:04:13,992][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2019-03-17T08:04:15,323][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x38daa0a4 run\>"}  
[2019-03-17T08:04:15,525][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
[2019-03-17T08:04:15,590][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
[2019-03-17T08:04:16,260][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
^C[2019-03-17T08:04:49,054][WARN][logstash.runner] SIGINT received. Shutting down.  
[2019-03-17T08:04:49,370][INFO][filewatch.observingtail] QUIT - closing all files and shutting down.  
{  
"nppri" =\> [  
[0] "0.00000"  
],  
"hard\_req\_queue" =\> [  
[0] "all.q"  
],  
"binding" =\> [  
[0] "NONE"  
],  
"ntix" =\> [  
[0] "0.00000"  
],  
"path" =\> "/usr/share/logstash/qstat.cache",  
"message" =\> "\<job\_list state="pending"\>\n \<JB\_job\_number\>140091542\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="arch" resource\_contribution="0.000000"\>lx\*\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</job\_info\>\n\</job\_info\>",  
"priority" =\> [  
[0] "0.00000"  
],  
"host" =\> "inn-elk-vm",  
"Job\_Number" =\> [  
[0] "140091542"  
],  
"@timestamp" =\> 2019-03-17T02:34:49.797Z,  
"Job\_State" =\> [  
[0] "pending"  
],  
"@version" =\> "1",  
"type" =\> "xml",  
"tags" =\> [  
[0] "multiline"  
]  
}  
[2019-03-17T08:04:51,264][INFO][logstash.pipeline] Pipeline has terminated {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x38daa0a4 run\>"}

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 17, 2019, 2:38am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/9 "2019-03-17T02:38:53Z")

</div>

also on writing to elasticsearch also only 1 tag data is written :

# {

"took" : 0,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 5,  
"successful" : 5,  
"skipped" : 0,  
"failed" : 0  
},  
"hits" : {  
"total" : 1,  
"max\_score" : 1.0,  
"hits" : [  
{  
"\_index" : "noida\_qstat-2019.03.17.22",  
"\_type" : "doc",  
"\_id" : "ext1iWkBTL48HKwyT\_vG",  
"\_score" : 1.0,  
"\_source" : {  
"binding" : [  
"NONE"  
],  
"Job\_State" : [  
"pending"  
],  
"@version" : "1",  
"type" : "xml",  
"tags" : [  
"multiline",  
"\_xmlparsefailure"  
],  
"host" : "inn-elk-vm",  
"path" : "/usr/share/logstash/qstat.cache",  
"Job\_Number" : [  
"140091542"  
],  
"priority" : [  
"0.00000"  
],  
"ntix" : [  
"0.00000"  
],  
"nppri" : [  
"0.00000"  
],  
"message" : "\<job\_list state="pending"\>\n \<JB\_job\_number\>140091542\</JB\_job\_number\>\n \<JAT\_prio\>0.00000\</JAT\_prio\>\n \<JAT\_ntix\>0.00000\</JAT\_ntix\>\n \<JB\_nppri\>0.00000\</JB\_nppri\>\n \<hard\_request name="arch" resource\_contribution="0.000000"\>lx\*\</hard\_request\>\n \<hard\_req\_queue\>all.q\</hard\_req\_queue\>\n \<job\_binding\>NONE\</job\_binding\>\n \</job\_list\>\n \</job\_info\>\n\</job\_info\>",  
"@timestamp" : "2019-03-17T02:22:38.974Z",  
"hard\_req\_queue" : [  
"all.q"  
]  
}  
}  
]  
}  
}

---

<div class="post-metadata">

### Author: ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)
#### Post date: [March 23, 2019, 1:46am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/10 "2019-03-23T01:46:07Z")

</div>

Hi , I was able to parse each line as a separate event by using below config, what I did was that I removed all the other tags and only kept \<job\_list\> tags , as my concerned data was inside those tags itself.

input  
{  
file  
{  
path =\> "/usr/share/logstash/qstat.cache"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
codec =\> multiline { pattern =\> "\</job\_list\>" what =\> "next" negate =\> true auto\_flush\_interval =\> 10 }  
}  
}  
filter  
{  
mutate { gsub =\> ["message", "\<job\_info._\>", ""] }  
mutate { gsub =\> ["message", "\<.xml version.\>", ""] }  
mutate { gsub =\> ["message", "\</job\_info._\>", ""] }  
mutate { gsub =\> ["message", "\<queue\_info._\>", ""] }  
mutate { gsub =\> ["message", "\</queue\_info._\>", ""] }  
xml  
{  
source =\> "message"  
remove\_namespaces =\> true  
store\_xml =\> false  
#target =\> "theXML"  
xpath =\> [  
"/job\_list/@state", "Job\_State",  
"/job\_list/JB\_job\_number/text()", "Job\_Number",  
"/job\_list/JAT\_prio/text()", "priority",  
"/job\_list/JAT\_ntix/text()", "ntix",  
"/job\_list/JB\_nppri/text()", "nppri",  
#"/job\_list/hard\_request[@name]", "hard\_request",  
"/job\_list/hard\_req\_queue/text()", "hard\_req\_queue",  
"/job\_list/job\_binding/text()", "binding"  
]  
}  
mutate {  
rename =\> [  
"[Job\_State][0]", "Job\_State",  
"[Job\_Number][0]", "Job\_Number",  
"[priority][0]", "priority",  
"[ntix][0]", "ntix",  
"[nppri][0]", "nppri",  
"[hard\_req\_queue][0]", "hard\_req\_queue",  
"[binding][0]", "binding"  
]  
}  
}  
output  
{  
stdout { codec =\> rubydebug }  
}

Thanks for you assistance all the way on this !

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 20, 2019, 1:46am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270/11 "2019-04-20T01:46:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.