How to deal with optionals fields in Logstash/ElasticSearch

[NicolasHennion (Nicolas Hennion (Nicolargo))]

have a problem when i try to import the following XML file in ElasticSearch with Logstash:

<test>
<param name="param1" raw="0x01">
<param name="param2" raw="0x02" eng="2">
<param name="param3" raw="0x03" eng="3">
</test>

My Logstash conf file is the following (extract):

xml {
    store_xml => false
    source => "message"
    xpath => [
        "/test/param/@name", "name",
        "/test/param/@raw", "raw",
        "/test/param/@eng", "eng"
    ]
}

After running Logstash, the ElasticSearch database contains:

"name": [
      "param1",
      "param2",
      "param3",
    ],
"raw": [
      "0x01",
      "0x02",
      "0x02",
    ],
"eng": [
      2,
      3,
    ],

I want this:

...
"eng": [
      null,
      2,
      3,
    ],

How can i force a null (or None or NaN) value in a non existing field ?

Thanks in advance !

[NicolasHennion (Nicolas Hennion (Nicolargo))]

I try to set the suppress_empty option to false but the result is the same...

Any idea ?

[CDR (Colton)]

Have you tried using a conditional statement for the empty fields? So say that eng is the field that is always empty

if [eng] == ""{mutate{update=>{"eng"=>"null"}}}

Or something along that line (code may vary)?

[NicolasHennion (Nicolas Hennion (Nicolargo))]

I already try but nothing change...

I think that after the xpath command the eng is not empty but equal to "eng": [2, 3]. So the code do not enter in the if statement.

[CDR (Colton)]

So if the field eng is not even in your log message then the if statement above needs to be changed. I am sorry I did not notice that at first. If if is not within the message but you still want it nulled in the filter section of the code try this:

if ![eng]{
   mutate{
      add_field => {"eng" => "null"}
   }
}