# Xml filter: create filter definition based on xsd
**URL:** https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581
**Category:** Logstash
**Created:** [July 25, 2018, 1:23pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581 "2018-07-25T13:23:35Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@JPelastic](https://discuss.elastic.co/u/JPelastic)
#### Post date: [July 25, 2018, 1:23pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/1 "2018-07-25T13:23:35Z")
Hi
Context: I need to put (several millions) XML invoices into ES and query and aggregate over their content.
Proposed solution: use Logstash xml filter plugin to map the XML to JSON
First question: For my context, would this be the best solution?
If so, second question: can I create the filter definition automatically from an XSD describing the invoices? Because the XSD contains about 500 possible fields and I will have to make about 10 different indices.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 25, 2018, 3:08pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/2 "2018-07-25T15:08:08Z")
Well if you need to use xpath, I suppose in theory you could use xslt to transform an xsd into a set of xpath expressions. But I would just use something like
```
xml {
source => "message"
store_xml => true
target => "theXML"
force_array => false
}
```
It will parse the XML and create a JSON structure that reflects it in a field called theXML
---
### Author:  [@JPelastic](https://discuss.elastic.co/u/JPelastic)
#### Post date: [July 25, 2018, 3:10pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/3 "2018-07-25T15:10:07Z")
Whaaaaaaaaaat? So simple?
That's awesome!
---
### Author:  [@JPelastic](https://discuss.elastic.co/u/JPelastic)
#### Post date: [July 25, 2018, 4:34pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/4 "2018-07-25T16:34:27Z")
So I started this up and added this config-file, without changing any other stuff in logstash configuration:
```auto
input{
file
{
path => "C:\elastic\XML4Logstash"
type => "xml"
}
}
filter{
xml{
source => "message"
store_xml => false
target => "orderrsp"
force_array => false
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
```
I was hoping logstash would now pick up any file in that folder?
I've been looking at the documentation. It says what exists, but not how it works.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 25, 2018, 5:41pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/5 "2018-07-25T17:41:26Z")
You almost certainly want store\_xml to be true.
The file input will read any files that match the path. So I would expect you to use
path =\> "C:\elastic\XML4Logstash\*.xml"
If you run logstash once it will read the files and store the fact that it has done so in the sincedb. If you restart logstash it will know it has read the files and start tailing them at the point it had read to so that if anything is appended it can process it. Probably not useful in your case. However, it will read any new files.
---
### Author:  [@JPelastic](https://discuss.elastic.co/u/JPelastic)
#### Post date: [July 26, 2018, 8:46am UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/6 "2018-07-26T08:46:02Z")
I got it doing stuff, but still it's hard to understand what is going on.
Please correct me if I'm wrong:
```auto
input{
file
{
path => "C:\elastic\XML4Logstash"
type => "xml"
}
}
filter{
xml{
source => "Orderrsp"
remove_namespaces => true
store_xml => true
target => "orderrsp"
force_array => false
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash"
}
}
```
Logstash reads xml from the inputfolder
Then filters it
- it takes the source and puts it in the message field. That field is an internal logstash field?
- it does some stuff to it an stores the result into the orderrsp field (again a logstash field).
next it will try to output it to ES in the logstash index (this is a very early proof of concept - no wildcards)
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 26, 2018, 12:35pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/7 "2018-07-26T12:35:57Z")
If you want to consume an entire file as a single event then you will need to use a multiline filter. There is an example [here](https://discuss.elastic.co/t/trying-to-parse-xml-logs/132749/2?u=badger). If you do not use a multiline filter then each line of the file will be a separate event.
The event will contain the contents of the file (either a single line, or the output of the multiline) in a field called "message", so that should be the source for you xml filter.
---
### Author:  [@JPelastic](https://discuss.elastic.co/u/JPelastic)
#### Post date: [July 26, 2018, 1:24pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/8 "2018-07-26T13:24:22Z")
Hi Badger
Thanks to your help I got it working all the way.
Only now I will need to define a mappings for my documents and that mapping is gonna be pretty large (like 100+ fields a piece).
Is there any way I can translate an XSD into an ES mapping on the fly?
Thanks
J.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 26, 2018, 3:08pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/9 "2018-07-26T15:08:21Z")
You might want to ask that in the elasticsearch forum.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 23, 2018, 3:08pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581/10 "2018-08-23T15:08:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.