Inconsistent XML - Conditional fields check

Elastic Stack Logstash
1

Hi,

I am importing an XML file from an external provider. The XML has many inconsistencies. Often fields vary. This means that I often index fields with incorrect data (I index the mapping expression).

I would like to be able to exclude some fields in case they are not presented in the XML and for other fields I want to insert a default value.

Example of field indexed with the mapping expression:
"registrationTypeId": "%{[registrationTypeId][0]}",
Preferred default value: 0

"efficiency": "%{[efficiency][0]}",
Preferred default value: 0

I have attempted to understand how to apply the conditional and drop from this thread, for the fields I wish to drop entirely: Conditional for empty string in field but it dont see where to apply it and how to treat it in the replace of mutate. Likewise with updating a fields value to a default value incase the value is not returned from the xpath.

My filter looks like this:

filter {
xml {
    #target => "doc"
    source => "message"
    store_xml => "false"
    xpath => ["//Statistik","statistic"]
    remove_namespaces => true
}

mutate {
    remove_field => ["message", "inxml", "xmldata"]
}

split {
    field => "statistic"
}

xml {
    source => "statistic"
    store_xml => "false"
    ...
    xpath => ["//Statistik/KoeretoejOplysningGrundStruktur/KoeretoejOplysningTypeAnmeldelseNummer/text()","registrationTypeId"]
    xpath => ["//Statistik/KoeretoejOplysningGrundStruktur/KoeretoejMotorStruktur/KoeretoejMotorKmPerLiter/text()","efficiency"]
    ...
}
mutate {
    replace => {
    	...
        "[vehicle][registrationTypeId]" => "%{[registrationTypeId][0]}"
        "[vehicle][efficiency]" => "%{[efficiency][0]}"
        ...
    }
}

mutate {
    ...
    remove_field => ["statistic"]
    remove_field => ["registrationTypeId"]
    remove_field => ["efficiency"]
    ...
}
}
2

I solved this by doing an if condition within filter, but outside mutate.

    if ![efficiency] {
        mutate {
            replace => {
                "[vehicle][efficiency]" => "Hello WOrld"
            }
        }
    }
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.