# Ingest xml file using Logstash

**URL:** <https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113>\
**Category:** Logstash\
**Created:** [November 11, 2020, 6:34pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113 "2020-11-11T18:34:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Priyanka3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyanka3/32/78835_2.png) [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\
**Post date:** [November 11, 2020, 6:34pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/1 "2020-11-11T18:34:19Z")

</div>

Hi,

I am very new to Elastic search and Logstash. But I am trying to ingest using Logstash.  
My xml file looks something like this

```
<control version='1'
<cust_details must_id='101'
cust_name="Stacy"
cust_city="Chicago"
cust_state="illinois </>
</control>

<prod_details prod_id="Prod123"
prod_name="Chips"
prod_aisle="B1"
prod_available="yes"
</prod_details>

<response trasaction_status="yes"
total amount="$1010"
</repsonse>

```

So this is what I have done so far

> input {  
> beats{  
> port=\>5044  
> }  
> fie{  
> path=\>"file path.xml"  
> start\_position=\>"beginning"  
> type="xml"  
> codex=\>multiline{  
> pattern=\>"^\<?control.\*\>"  
> negate="true"  
> what="previous"  
> auto\_flush\_interval=\>1  
> max\_lines=\>3000  
> }  
> }  
> }  
> output{  
> Elasticsearch{  
> hosts=\>["localhost:9200"]  
> index=\>"Logstash"  
> }  
> }

I don't know how to process.

Anyone can help?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 11, 2020, 6:46pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/2 "2020-11-11T18:46:39Z")

</div>

Please edit you post, select the XML, and click on \</\> in the toolbar above the edit panel. That will change the display from

\<cust\_details must\_id="101"  
cust\_name="Stacy"  
cust\_city="Chicago"  
cust\_state="illinois \</\>

to

```
<cust_details must_id="101"
cust_name="Stacy"
cust_city="Chicago"
cust_state="illinois </>

```

then do the same for the logstash configuration.

The codex option should be codec.

Your pattern appears to be wrong. If you want to combine all the lines that follow the `<control version="1"` line up until the next occurrence of that pattern then use

```
pattern => "^<control"

```

The values for `negate` and `what` look OK.

Note that your prod\_details element is not valid XML. If it really looks like that you will get parse failures.

---

<div class="post-metadata">

**Author:** ![Priyanka3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyanka3/32/78835_2.png) [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\
**Post date:** [November 11, 2020, 7:23pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/3 "2020-11-11T19:23:15Z")

</div>

Hi,

Can you give me a general idea on how to do it? I can't paste the actual xml file due to sensitivity of the file. But it pretty much is like the example I gave. It have different tags and various values in them.

Any leads would be helpful. There are no videos or links for proper parsing of xml data to Elastic search. So any leads you can give me now, I'll take it

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 11, 2020, 8:00pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/4 "2020-11-11T20:00:09Z")

</div>

If you want to consume the entire file as a single event then you can do something like [this](https://discuss.elastic.co/t/append-metadata-to-every-event-in-xml-file-being-ingested-to-elasticsearch/165914/2).

If your file contains multiple XML documents you must consume them separately. If they all start with `<control` then your multiline codec should be

```
codec => multiline {
    pattern=>"^<control"
    negate => "true"
    what => "previous"
    auto_flush_interval => 1
    max_lines => 3000
}

```

If you want to parse the entire message then just use

```
xml { source => "message" store_xml => true target => "theXML" }

```

If you need specific elements from the document you can use the xpath option instead of setting store\_xml.

The xml filter typically just works. The hard part is tweaking the multiline codec so that each event contains a complete XML document.

---

<div class="post-metadata">

**Author:** ![Priyanka3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyanka3/32/78835_2.png) [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\
**Post date:** [November 11, 2020, 8:09pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/5 "2020-11-11T20:09:41Z")

</div>

is there a video or a document I can follow? just to understand the different field names and the values we can give in them

---

<div class="post-metadata">

**Author:** ![Priyanka3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyanka3/32/78835_2.png) [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\
**Post date:** [November 11, 2020, 8:29pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/6 "2020-11-11T20:29:26Z")

</div>

Considering my previous xml file, I would like to get different fields within each tag  
say for example, we have the cust\_details. I want to extract the different fields available in them - cust\_name, cust\_city, cust\_state.  
How do I achieve this

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 11, 2020, 8:46pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/7 "2020-11-11T20:46:24Z")

</div>

Use

`xml { source => "message" store_xml => true target => "theXML" }`

If you then need to move all the fields to the top level use a ruby filter, like [this](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006/2).

---

<div class="post-metadata">

**Author:** ![Priyanka3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priyanka3/32/78835_2.png) [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\
**Post date:** [November 11, 2020, 11:28pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/8 "2020-11-11T23:28:30Z")

</div>

what is "target" in the above syntax?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 12, 2020, 12:07am UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/9 "2020-11-12T00:07:21Z")

</div>

The [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html#plugins-filters-xml-target) covers that.

---

<div class="post-metadata">

**Author:** ![enrique.villar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/enrique.villar/32/77925_2.png) [@enrique.villar](https://discuss.elastic.co/u/enrique.villar)\
**Post date:** [November 12, 2020, 8:29pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/10 "2020-11-12T20:29:55Z")

</div>

In the Logstash configuration I can see beats as the input.  
You need to install and configure filebeat in your server to send the xml to Logstash.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 10, 2020, 8:30pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113/11 "2020-12-10T20:30:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.