Date filter can't access nested field from jdbc input

Elastic Stack Logstash
1

I have a

input {
  jdbc { 
    target => "DataRow"
  }
}
filter {
  mutate {
     add_field => { "rubentest" => "%{[DataRow][created_at]}" }
  }
  date {
    match => [ "rubentest", "ISO8601"]
    target => "rubentest2"
  }
  date {
    match => [ "[DataRow][created_at]", "ISO8601"]
    target => "rubentest3"
  }

}

The date filter fails for [DataRow][created_at] (_dateparsefailure) , but if I copy the field [DataRow][created_at] to [rubentest] and use date on that it works.

I'm guess I'm using the wrong syntax to access nested field in date filter but I don't understand what I'm doing wrong.

2

I wonder if this has something to do with the jdbc input because I tried with the generator input plugin instead of the jdbc input plugin and I there I can parse the field ok:

input {
  generator {
    count => 2
    ecs_compatibility => "v8"
    message =>  "test"
    add_field => {
      "[DataRow][created_at]" => "2024-12-12T10:29:57.832Z"
    }
  }
}
filter {
  date {
    match => ["[DataRow][created_at]", "ISO8601"]
    target => "rubentest3"
  }
}
output {
  stdout {codec => json_lines}
}


{"TableauAuditLog":{"created_at":"2024-12-12T10:29:57.832Z"},"@timestamp":"2024-12-12T10:29:57.832Z","host":{"name":"xxxx"},"@version":"1","message":"test","event":{"original":"test","sequence":1}}

I suspect that the when I'm using the jdbc input the [DataRow][created_at] is not a string it must be represented some other way and it's only printed as "2024-12-12T10:29:57.832Z" when it's serialized in JSON lines.

3

So now I just use mutate instead of date like this

  mutate {
    copy => { "[DataRow][created_at]" => "@timestamp"}
  }

I can't find any description of the jdbc input plugin maps the timestamp/datetime columns in jsql to logstash, but I'm the [DataRow][created_at] is a "Logstash Timestamp object" and not a string. And logstash knows how to convert the Logstash::Timestamp object to a string at the output.

4

That is correct. The jdbc input automatically converts datetime columns to LogStash::Timestamp objects. You would see a difference when using a rubydebug codec on the output. Instead of "2024-12-12T10:29:57.832Z" a LogStash::Timestamp would be 2024-12-12T10:29:57.832Z, without quotes.

1 Like