# How i can create index on xml tags

**URL:** <https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175>\
**Category:** Logstash\
**Created:** [February 27, 2019, 1:37pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175 "2019-02-27T13:37:07Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [February 27, 2019, 1:37pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/1 "2019-02-27T13:37:08Z")

</div>

Hello,

I am new to logstash, i have one requirement like below.

i want to create index on xml tag. this xml is present in database table. I am able to index on column which having this xml. but the requirement is to index on particular xml tags.

could you please help me with example

Xml from db table column as below

\<?xml version="1.0" encoding="UTF-8"?\>

```
<alert-header>
    <elem name="alertDate">2019-01-10 01:56:43</elem>
    <elem name="score">100</elem>
    <elem name="alertEntityKey">1539912_029_07/01/2018 </elem>
    <elem name="partyType">Entity</elem>
    <elem name="partyYOB"/>
    <elem name="partyBirthLocation"/>
    <elem name="ahData">
        <elem name="alertDate">2019-01-10 01:56:43</elem>
    </elem>
    <elem name="ahData">
        <elem name="jobID">01-10-2019</elem>
    </elem>
    <elem name="ahData">
        <elem name="jobName">TEST_PID</elem>
    </elem>
    <elem name="ahData">
        <elem name="jobType">LARGEBATCH</elem>
    </elem>
    <elem name="ahData">
        <elem name="score">100</elem>
    </elem>
    <elem name="ahData">
        <elem name="numberOfHits">7</elem>
    </elem>
    <elem name="ahData">
        <elem name="partyKey">1539912_029_07/01/2018</elem>
    </elem>
    <elem name="ahData">
        <elem name="partySourceId"/>
    </elem>
    <elem name="ahData">
        <elem name="partyName">ISIS IN THE ISLAMIC SAHEL</elem>
    </elem>
    <elem name="ahData">
        <elem name="partyLName">ISIS IN THE ISLAMIC SAHEL</elem>
    </elem>
    <elem name="ahData">
        <elem name="partyAliases"/>
    </elem>
    <elem name="ahData">
        <elem name="alertType">Sanctions</elem>
    </elem>
    <partyIds/>
    <elem name="partyNatCountries">
        <elem name="countryCd"/>
    </elem>
    <elem name="partyAddresses">
        <elem name="partyAddressLine1"/>
        <elem name="partyAddressLine2"/>
        <elem name="partyCity"/>
        <elem name="partyPostalCd"/>
        <elem name="partyStateProvince"/>
        <elem name="countryCd"/>
    </elem>
</alert-header>

```

i want to index on jobId, jobName etc...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 27, 2019, 2:59pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/2 "2019-02-27T14:59:46Z")

</div>

You can parse the XML using

```
xml { source => "message" target => "[@metadata][XML]" store_xml => true }

```

The resulting XML will look like this

```
           "XML" => {
    "elem" => [
        [0] {
               "name" => "alertDate",
            "content" => "2019-01-10 01:56:43"
        },
        [1] {
               "name" => "score",
            "content" => "100"
        },
[...]
        [6] {
            "name" => "ahData",
            "elem" => [
                [0] {
                       "name" => "alertDate",
                    "content" => "2019-01-10 01:56:43"
                }
            ]
        },

```

You can use a ruby filter to iterate over the array, and if the array entry has name and content fields use them to add a field to the event, and if the array entry has a elem field do the same check on that. Something like this:

```
    ruby {
        code => '
            event.get("[@metadata][XML][elem]").each { |x|
                if x["name"] and x["content"]
                    event.set(x["name"], x["content"])
                else
                    if x["elem"].kind_of?(Array)
                        x["elem"].each { |y|
                            if y["name"] and y["content"]
                                event.set(y["name"], y["content"])
                            end
                        }
                    end
                end
            }
        '
    }

```

Then you may need special handling for some of the fields, but this should get you started.

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [February 27, 2019, 3:17pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/3 "2019-02-27T15:17:55Z")

</div>

Thanks for response.

My current logstash-config.conf is as below

input {  
jdbc {  
#input Configuration  
jdbc\_connection\_string =\> "jdbc:oracle:thin:@oraasgtd37-scan.nam.nsroot.net:8889/SID"  
jdbc\_user =\> "admin"  
jdbc\_password =\> "\*\*\*\*\*_"  
jdbc\_driver\_library =\> "I:\Jars\ojdbc6.jar"  
jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver"  
statement =\> "select html\_file\_key from alerts where deleted =0"  
 #use\_column\_value =\> true  
 #tracking\_column =\> "alert\_internal\_id"  
 #schedule =\> "_ \* \* \* \* \*"

```
	}

```

}

output {  
elasticsearch {  
#output configuration  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "alert\_index"  
document\_type =\> "alert"  
#document\_id =\> "%{alert\_internal\_id}"  
}  
stdout{  
codec =\> rubydebug  
}  
}

what changes required in this to parse the xml.

Note: HTML\_FILE\_KEY returns the xml

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 27, 2019, 3:38pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/4 "2019-02-27T15:38:40Z")

</div>

Add

```
filter {
    xml { source => "html_file_key" target => "[@metadata][XML]" store_xml => true }
    ruby {
        code => '
            event.get("[@metadata][XML][elem]").each { |x|
                if x["name"] and x["content"]
                    event.set(x["name"], x["content"])
                else
                    if x["elem"].kind_of?(Array)
                        x["elem"].each { |y|
                            if y["name"] and y["content"]
                                event.set(y["name"], y["content"])
                            end
                        }
                    end
                end
            }
        '
    }
}
```

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [February 28, 2019, 6:33am UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/5 "2019-02-28T06:33:42Z")

</div>

I added this filter as it is, but i am getting Error.

[ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass

Please help me out on this, i am unaware of ruby.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 28, 2019, 2:03pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/6 "2019-02-28T14:03:15Z")

</div>

I suggest changing the output to be

```
stdout { codec => rubydebug { metadata => true } }

```

and see of the XML was successfully parsed to include [@metadata][XML][elem]

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [February 28, 2019, 4:23pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/7 "2019-02-28T16:23:34Z")

</div>

Thank you so much!  
I am able to parse xml now. But my requirement is to search with name, How i can right the uri to get alertDate or score from this parsed xml. could you please help me on this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 28, 2019, 4:36pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/8 "2019-02-28T16:36:49Z")

</div>

That is what the ruby filter does.

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [February 28, 2019, 4:50pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/9 "2019-02-28T16:50:31Z")

</div>

But how i get the content with respect to name in elastic search

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [March 6, 2019, 8:08am UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/10 "2019-03-06T08:08:32Z")

</div>

Any update on this Sir

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 6, 2019, 12:52pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/11 "2019-03-06T12:52:34Z")

</div>

What exactly do you not like about the events created by the ruby filter. Please show an event and what you want to change in it.

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [March 6, 2019, 1:06pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/12 "2019-03-06T13:06:43Z")

</div>

Hi Badger, actually i am still struggling to add fields(name=\>content) to elastic search. I am not aware of ruby.

---

<div class="post-metadata">

**Author:** ![mandar.raj](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@mandar.raj](https://discuss.elastic.co/u/mandar.raj)\
**Post date:** [March 6, 2019, 2:09pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/13 "2019-03-06T14:09:57Z")

</div>

Is there any other way using XPATH??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 3, 2019, 2:10pm UTC](https://discuss.elastic.co/t/how-i-can-create-index-on-xml-tags/170175/14 "2019-04-03T14:10:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.