# Storing logs containing several different XML types/schemas into ES

**URL:** https://discuss.elastic.co/t/storing-logs-containing-several-different-xml-types-schemas-into-es/207764
**Category:** Elasticsearch
**Created:** [November 13, 2019, 6:59pm UTC](https://discuss.elastic.co/t/storing-logs-containing-several-different-xml-types-schemas-into-es/207764 "2019-11-13T18:59:49Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![zenluiz](https://avatars.discourse-cdn.com/v4/letter/z/dc4da7/32.png) [@zenluiz](https://discuss.elastic.co/u/zenluiz)
#### Post date: [November 13, 2019, 6:59pm UTC](https://discuss.elastic.co/t/storing-logs-containing-several-different-xml-types-schemas-into-es/207764/1 "2019-11-13T18:59:49Z")

</div>

Hi,  
I have millions of lines of logs that I want to index in ES, using Filebeat + Logstash.  
Each log line can have a different XML that represents a SOAP message that has been received by a WebService and logged, like in this example:

```
2019-11-13 10:00:1234 <?xml><Envelope><Body><GetUnitByPosition><Position>1234</Position></GetUnitByPosition></Body></<Envelope>
2019-11-12 09:30:5678 <?xml><Envelope><Body><GetPositionByName><Name>Position1</Name></GetPositionByName></Body></<Envelope>
2019-11-11 08:30:5678 <?xml><Envelope><Body><UpdatePosition><Position>1234567</Position><Name>Position2</Name><Tag>9876</Tag></UpdatePosition></Body></<Envelope>

```

In the real case, there are around 100 possible XML messages received by this web service, so I wanted to make the logs easier to read by storing the XML, using **store\_xml =\> true** option in the xml filter in Logstash. Also, I am using **XPath** to extract every single possible element from the XMLs (yes, a heck of a tedious job) into separate fields.

Problems are:

1. Creating XPath for each element is too much work, but more than that, if there is any change on any of the XML schemas (or new elements), I would need to change the logstash config as well. I would like to avoid that.

2. To avoid that, I thought on removing the XPaths and work with the fields that the XML filter plugin generates. However, doing that brings another problem: since there are too many combinations / elements in XMLs, I get many error messages saying the limit of 1000 fields has been reached. That's because it generates the following fields for the example I gave above:

So, the question is: does anyone have any experience with a situation similar to this one? Any recommendation? Maybe I am missing something, a configuration to limits, etc.

Thanks!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 11, 2019, 6:59pm UTC](https://discuss.elastic.co/t/storing-logs-containing-several-different-xml-types-schemas-into-es/207764/2 "2019-12-11T18:59:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
