Logstash: Adding custom field to documennt

Elastic Stack Logstash
1

Hi all.
I have the following XML file format:

     <?xml version="1.0" encoding="UTF-8"?>
<Object id="1" MUID="XXXX" GUID="XXXXXX2">
	<Attribute name="Others" value="---"/>
	<Attribute name="Module name" value="Anforderungen_von_AbC_und_AbP_an_ORC"/>
	<Attribute name="fullName" value="xxxxxxxxxxxx"/>
	<Attribute name="ModuleUniqueID" value=""/>
	<Attribute name="Last Modified On" value="04/16/18 16:57:49"/>
	<Attribute name="ModuleType" value="CRS"/>
	<Attribute name="Current Date" value="06/29/18 06:47:55"/>
</Object>
<Object id="1" MUID="XXXX" GUID="XXXXXX3">
	<Attribute name="Others" value="---"/>
	<Attribute name="Module name" value="Anforderungen_von_AbC_und_AbP_an_ORC"/>
	<Attribute name="fullName" value="xxxxxxxxxxxx"/>
	<Attribute name="ModuleUniqueID" value=""/>
	<Attribute name="Last Modified On" value="04/16/18 16:57:49"/>
	<Attribute name="ModuleType" value="CRS"/>
	<Attribute name="Current Date" value="06/29/18 06:47:55"/>
</Object>

I have liked to give a custom ID to each of this object while indexing it into elasticsearch. The ID should be a combination of the Last Modified On + Object GUID. I'm not the Logstash expert, so it's why i'm asking here if some one can give me a small tip on how to realise it. I'm pretty sure it can be made with the Ruby plugin or the XML plugin but not sure what exactly to do.

PS: The attributes are very dynamic, the order is not always the same.

Cheers.

2

What does the XML look like?

3

I Updated the question. Somehow the XML content was not displayed!

4

I would use a multiline codec for that

input { stdin { codec => multiline { pattern => "</Object>" negate => true what => "next" auto_flush_interval => 3 } } }

Then an xml filter

filter { xml { source => "message" store_xml => true target => "theXML" force_array => false } }

At that point you have a couple of choices. You could flatten the attributes using a ruby filter.

    ruby {
        code => '
            event.get("[theXML][Attribute]").each { |v|
                event.set("[Attributes]" + v["name"], v["value"])
            }
        '
    }

Or you could iterate over the set and just stash the value you want in a temporary place, then use it to build the id.

    ruby {
        code => '
            event.get("[theXML][Attribute]").each { |v|
                if v["name"] == "Last Modified On"
                    event.set("[@metadata][lastModified]" , v["value"])
                end
            }
        '
    }
1 Like
5

Thanks for your answer. I will try this. It looks like it's what I was searching for!

6

It works like a charm!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.