Problem with "split" and "add_field" in mutate filter

Elastic Stack Logstash
1

Hello,

I have the following log line :

"1","O","I","191118 190923","E","0","1455","SFTP","PNVIO111","IT9","/data/files/TRANS","FOPIT901-9281025"

And the following dissect filter :

dissect {
mapping => {
"message" => '"%{type}","%{direction}","%{mode}","%{date}","%{status}","%{code}","%{size}","%{protocol}","%{src}","%{dst}","%{path}","%{file_component}"'
}

My final goal is to split the last field "file_component" into two new fields ("shortidf" and "trans_id"), based on the "-" splitting char.

So I wrote thess mutate filters (based on the doc example here Mutate filter plugin | Logstash Reference [8.11] | Elastic) :

mutate {
copy => { "file_component" => "idf" }
}

      mutate {
      split => ["idf", "-"]
      add_field => { "shortidf" => "%{idf[0]}" }
      }

The "copy" is OK (I obtain a field called "idf" in my json output)
the "split" is OK (if I comment the add_field part), I obtain :

"file_component" => "FOPIT901-9281025",
"idf" => [
[0] "FOPIT901",
[1] "9281025"
],

But the add_field => { "shortidf" => "%{idf[0]}" } part gives me an error on logstash output if I activate it in the mutate filter :

org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: idf[0]

Any idea on what I'm doing wrong ?

2

Use [idf][0]

2 Likes
3

Great ! thank you.
It's ok now.

What have I missed ? on the documentation, it's clearly wrote as this :
add_field => { "shortHostname" => "%{hostname[0]}" }

If I can abuse of your help, what would be the best method to split the field shortidf, containing "FOPIT901" into two fields containing :
"source" = 3 first chars of the shortidf field (FOP)
"target" = 3 next chars (IT9)

Any idea ?

The split method uses a separator (but there isn't here)
The truncate method uses a length_bytes but it's not possible to truncate into multiple fields ?

4

In older releases that would work, but recently the code was change to disallow ambiguous field references like hostname[0]. Sadly the documentation has not been updated to reflect that.

You can use mutate+gsub to capture fixed length substrings.

mutate { copy => { "shortidf" => "source" "shortidf" => "target" } }
mutate {
    gsub => [
        "source", "^(.{3}).*", "\1",
        "target", "^.{3}(.{3}).*", "\1"
    ]
}
5

Wow, really impressed...
Thanks for your precious help.

Regards,
Xavier

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.