# Can i use file filter for xml docs

**URL:** <https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913>\
**Category:** Logstash\
**Created:** [June 4, 2015, 9:32am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913 "2015-06-04T09:32:21Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 4, 2015, 9:32am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/1 "2015-06-04T09:32:21Z")

</div>

input {  
file {  
path =\> "D:\folder\*.xml"  
type =\> "string"  
start\_position =\> "beginning"  
}  
}

filter {  
xml {....  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 4, 2015, 9:58am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/2 "2015-06-04T09:58:54Z")

</div>

There is no file filter so I'm assuming you're talking about the file input. Yes, this works but unless you use the multiline filter or codec to join the whole file (i.e. all _physical_ lines) into a single message (one _logical_ line) the xml filter won't work.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 4, 2015, 10:16am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/3 "2015-06-04T10:16:08Z")

</div>

I have nested xml like this....

"\<?xml version="1.0" encoding="UTF-8"?\>

\<?Status\>200 \<?response\> \<?record1\> \<?data1\>a.. \<?data2\>a.. \<?data3\>a.. \<?record1\> : \<?record N\>... \<?record N\> " I wish to parse from \<?response\> and require to output every \<?record.\> as new line.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 4, 2015, 1:57pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/4 "2015-06-04T13:57:09Z")

</div>

Any suggestion for handling this?

---

<div class="post-metadata">

**Author:** ![rafaltrojniak](https://avatars.discourse-cdn.com/v4/letter/r/ecc23a/32.png) [@rafaltrojniak](https://discuss.elastic.co/u/rafaltrojniak)\
**Post date:** [June 4, 2015, 2:17pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/5 "2015-06-04T14:17:52Z")

</div>

Just like 'magnusbaeck' said - this will work, but you have to use multiline filter.

Because 'file' input is using each line as separate event, you have to join the events using multiline plugin, and than use xml plugin to transform XML to separate fields.

Here is a pretty good example of that

> <https://stackoverflow.com/questions/25668970/cant-parse-xml-input-with-logstash-filter>

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 4, 2015, 2:25pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/6 "2015-06-04T14:25:20Z")

</div>

Thanks! i didnt understand what 'magnusbaeck' illustrated earlier. By this example it is more clear now.  
Do we need to use multiline filter for JSON as well?

---

<div class="post-metadata">

**Author:** ![rafaltrojniak](https://avatars.discourse-cdn.com/v4/letter/r/ecc23a/32.png) [@rafaltrojniak](https://discuss.elastic.co/u/rafaltrojniak)\
**Post date:** [June 4, 2015, 9:34pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/7 "2015-06-04T21:34:29Z")

</div>

This is depending on how the message is written. They both (JSON and XML) could be written one per line, or one in many lines.

For example

- XML in one line :  
`<test><some_field>value<some_field></test>`
- XML in multiple lines  
`<test> <some_field>value</some_field> </test>`

When the single event (Written in JSON or XML - that doesn't matter) i swritten in multiple lines - that generates multiple logstash events - you have to join them to single event using multiline filter.  
If whole event is written in single line - it is already single Logstash event - you do not have to do it.  
That all is just a consequence of assumption, that single event is single file line.

Regards,

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 5, 2015, 11:20am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/8 "2015-06-05T11:20:01Z")

</div>

Logstash config: Not parsing xml...

input {  
file {  
path =\> "/opt/Log/\*.xml"  
type =\> "test"  
}  
}

filter {  
multiline {  
pattern =\> "^\s\s(\s\s(\s\s|\</data\>))"  
what =\> "previous"  
}

```
xml {
        target => "data"
        source => "message"
        add_field => {
            "testId" => "%{[data][testId]}"
        "loadTime"=> "%{[data][average][0][loadTime]}"
        "TTFB"=> "%{[data][average][0][TTFB]}"
            
        }
    }

```

}

output {  
elasticsearch {  
action =\> "index"  
host =\> "172.27.155.109"  
index =\> "xml1"  
workers =\> 1  
}  
stdout {}  
}

XML Input:

\<?xml version="1.0" encoding="UTF-8"?\> \<?response\> \<?statusCode\>200 \<?data\> \<?testId\>150603\_VD\_9VJ \<?average\> \<?loadTime\>3594 \<?TTFB\>282

---

<div class="post-metadata">

**Author:** ![rafaltrojniak](https://avatars.discourse-cdn.com/v4/letter/r/ecc23a/32.png) [@rafaltrojniak](https://discuss.elastic.co/u/rafaltrojniak)\
**Post date:** [June 5, 2015, 11:52am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/9 "2015-06-05T11:52:02Z")

</div>

It looks to me like your multiline configuration is not proper.

Please look into my example where :

- First line (`<?xml...`) is dropped
- XML is joined into the whole `<response>` block
- XML data are extracted

Hope that helps

Rules :

> <https://github.com/rafaltrojniak/logstash_rules/blob/xm_multiline/rules/xml_multiline.conf>

  
Documentation:  

> <https://github.com/rafaltrojniak/logstash_rules/blob/xm_multiline/doc.md>

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 5, 2015, 12:11pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/10 "2015-06-05T12:11:34Z")

</div>

input {  
file {  
path =\> "/opt/Log/\*.xml"  
type =\> "test"  
}  
}

filter {  
if [message] =~ "^\<?xml .\*" {  
drop {}  
}  
multiline {  
pattern =\> "^\</response\>"  
negate =\> true  
what =\> "next"  
}

xml {  
source =\> "message"  
target =\> "data"  
}  
}

output {  
stdout {}  
}

I tried with above config. But No luck. Am I still missing anything.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 5, 2015, 12:42pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/11 "2015-06-05T12:42:48Z")

</div>

when i run in debug mode I get:

Flushing {:plugin=\>\<LogStash::Filters::Multiline pattern=\>"^\<\/response\>", what=\>"next", source=\>"message", stream\_identity=\>"%{host}.%{path}.%{type}"\>, :level=\>:debug, :file=\>"(eval)", :line=\>"43", :method=\>"initialize"}  
\_discover\_file\_glob: /opt/Log/\*.xml: glob is: ["/opt/Log/XML\_webpaeSu.xml"] {:level=\>:debug, :file=\>"filewatch/watch.rb", :line=\>"132", :method=\>"\_discover\_file"}

without any exception

---

<div class="post-metadata">

**Author:** ![rafaltrojniak](https://avatars.discourse-cdn.com/v4/letter/r/ecc23a/32.png) [@rafaltrojniak](https://discuss.elastic.co/u/rafaltrojniak)\
**Post date:** [June 5, 2015, 1:11pm UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/12 "2015-06-05T13:11:56Z")

</div>

Hmm. I don't know what is wrong here.  
Can you push the file through the logstash using no filters, and display the JSON results ? Maybe there is something wrong with the input plugin or the file itself..

Please escape the output properly using formatting, so I can read it easier and less error-prone.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:38am UTC](https://discuss.elastic.co/t/can-i-use-file-filter-for-xml-docs/1913/13 "2017-07-06T05:38:25Z")

</div>